Anthropic has barred users from exhibiting “sustained and needless abusive or cruel behavior” toward its models, as the company’s leaders continue to ponder machine consciousness.
The Verge first reported the change in policy.
A spokesperson for Anthropic, which is behind AI chatbot Claude, did not immediately respond to an inquiry about what could be considered “abusive or cruel”.
In an online user policy, the San Francisco-based company noted that its ban would not apply to common user frustrations, model testing or “dark creative themes”.
Anthropic’s usage policy previously contained restrictions on abusive conduct.
The company’s large language models have the ability to end a conversation if a user is being persistently harmful. When that feature rolled out last August, the company framed it as a safeguard for AI’s welfare.
“We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future. However, we take the issue seriously, and alongside our research program we’re working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible. Allowing models to end or exit potentially distressing interactions is one such intervention,” according to its website.
The notion of AI systems being conscious has been highly polarizing in and outside the tech world.
Anthropic CEO Dario Amodei has said he cannot rule out the possibility.
Meanwhile, Sam Altman, CEO of rival OpenAI, has appeared averse to the idea.
“I am very uncomfortable about people trying to ascribe religious force or a surrender of human judgment to AI models, and think it is a real safety issue,” he wrote in an X post, days after the New York Times reported on Anthropic leaders’ extensive conversations with religious scholars.