Anthropic Says Claude Can End Chats Over Extreme Abuse
Anthropic has updated its Claude usage policy to prohibit sustained, needless abusive behavior toward its models in extreme cases. Ordinary frustration…
Anthropic has changed the rules for Claude AI users: repeated, needless cruelty toward its models may lead the chatbot to end a conversation. The policy applies only in extreme cases, while frustration, model testing and dark creative themes remain permitted.
The company announced the change on Thursday. It has already sparked debate on Reddit, in blogs and elsewhere about whether companies should regulate how people behave toward software.
What the Claude AI policy allows
Anthropic’s blog post describes the restriction as addressing abuse toward its models. It targets “sustained and needless abusive or cruel behavior” carried out “with no discernible purpose,” the company said.
There is room to argue. Users can express frustration, push back against a model, explore dark themes in creative work, and test or research the system. Anthropic says it would enforce the rule only as a last resort; in those cases, Claude will end the conversation.
The company has not spelled out exactly which exchanges would trigger that response. An earlier Anthropic post, published in August 2025, discussed Claude’s “aversion to harm” and situations in which the model might shut down a chat. Its examples included requests for sexual content involving minors and attempts to obtain information that could enable large-scale violence or acts of terror.
Anthropic said that when Claude Opus 4 encountered harmful content of this kind, it showed “a pattern of apparent distress.” The account leaves a key distinction unresolved: whether the company is responding to harmful interaction patterns as a safety measure, or making a claim about what the model experiences.
Anthropic says model welfare remains uncertain
Asked by CNET what prompted the policy change, an Anthropic spokesperson said the company had not reached a conclusion about whether models can be harmed. “We’re uncertain whether models can experience harm, and we continue to explore this question in our research on model welfare, but we also believe that taking Claude’s interests and potential welfare into account may be relevant to safety,” the spokesperson said.
That uncertainty is central to the debate. Chatbots use human-like language, including apologies and expressions of concern, which can lead users to treat them as if they have feelings. The same conversational qualities can make a chatbot seem like a trusted collaborator, even though it is software.
Anthropic has argued that models need to handle emotionally difficult situations to be reliable and safe. In a paper earlier this year, the company wrote: “Even if [the models] don’t feel emotions the way that humans do, or use similar mechanisms as the human brain, it may in some cases be practically advisable to reason about them as if they do.”
Anthropic CEO Dario Amodei has also left open the question of machine consciousness. In a February interview with The New York Times, he said he was “open” to the possibility that AI models were conscious.
Keith Kakadia, founder and CEO of Sociallyin, which has studied Claude’s growth, said describing user behavior as “cruel” raises broader questions about whether a product can be hurt. The word carries emotional weight, he said, and suggests the chatbot has feelings and boundaries.
For users, the practical change is narrow but clear: they can challenge Claude and test its limits, but Anthropic says the model may end a conversation after repeated abusive behavior with no discernible purpose.
Source: cnet.com
