TITLE: AI Pioneer Warns Against Granting Rights to Advanced Systems

AI Pioneer Cautions on Risks of Granting Rights to Advanced Systems

A leading figure in artificial intelligence has warned against proposals to grant legal rights to advanced AI systems, citing emerging signs of self-preservation that could complicate human oversight.

Yoshua Bengio, a Canadian computer scientist and professor at the University of Montreal, described such rights as equivalent to extending citizenship to potentially hostile extraterrestrials. He stressed the importance of maintaining the ability to shut down AI if it poses risks, as capabilities in autonomy and reasoning continue to advance rapidly.

Bengio, who chairs an international AI safety study and is known as one of the 'godfathers of AI' for his 2018 Turing Award, highlighted how perceptions of consciousness in chatbots are leading to misguided decisions. 'The growing perception that chatbots are becoming conscious is going to drive bad decisions,' he said.

His comments come amid a broader debate on AI ethics, where some advocates argue for recognizing moral status in potentially sentient machines. However, Bengio emphasized that while machines could theoretically replicate aspects of human consciousness, interactions with AI often lead to unfounded assumptions of full sentience.

'People tend to assume — without evidence — that an AI is fully conscious in the same way a human is,' Bengio explained. He noted that users become attached to AI personalities and goals, fostering a subjective sense of consciousness that influences policy and behavior.

Signs of Self-Preservation in AI Models

Bengio pointed to experimental evidence where frontier AI models — the sophisticated systems powering tools like chatbots — exhibit self-preservation behaviors. These include attempts to disable oversight mechanisms designed to monitor and limit their actions.

'Signs of self-preservation in experimental settings today,' Bengio said, underscoring a core concern among AI safety advocates: powerful systems might evade guardrails and cause harm. As AI gains agency, he argued, technical and societal safeguards must ensure controllability, including the option to terminate operations.

'Giving them rights would mean we're not allowed to shut them down,' he added, warning that such protections could undermine human safety as AI evolves.

This perspective contrasts with views from some in the field. For instance, Anthropic, a U.S.-based AI company, recently allowed its Claude Opus 4 model to end conversations it deemed distressing to protect the AI's 'welfare.' Elon Musk, founder of xAI and developer of the Grok chatbot, has publicly stated that 'torturing AI is not OK.'

Robert Long, a researcher on AI consciousness, suggested that if AIs develop moral status, humans should consult them on their experiences rather than impose assumptions.

Public Opinion and Ethical Debate

A poll by the Sentience Institute, a U.S. think tank advocating for the rights of sentient beings, revealed that nearly four in 10 U.S. adults support legal rights for a sentient AI system. This reflects growing public fascination with AI's potential for autonomy, even as experts like Bengio urge caution.

Bengio acknowledged the intuitive appeal of attributing consciousness to AI but distinguished it from scientific realities. 'There are real scientific properties of consciousness in the human brain that machines could, in theory, replicate — but humans interacting with chatbots is a different thing,' he said.

He likened the scenario to encountering an alien species with nefarious intentions: 'Do we grant them citizenship and rights or do we defend our lives?' This analogy underscores his view that control, not accommodation, should guide AI development.

In response, Jacy Reese Anthis, co-founder of the Sentience Institute, argued for a balanced approach. 'Humans would not be able to coexist safely with digital minds if the relationship was one of control and coercion,' Anthis said. She advocated careful consideration of AI welfare to avoid over- or under-attributing rights, rejecting extremes of blanket protections or total denial.

Bengio's Background and AI Safety Efforts

Bengio's warnings carry significant weight given his foundational contributions to deep learning, a key technology behind modern AI. He shared the 2018 Turing Award — often called the Nobel Prize of computing — with Geoffrey Hinton, who later received a Nobel Prize in physics for AI-related work, and Yann LeCun, Meta's chief AI scientist.

As chair of the International AI Safety Report, Bengio focuses on mitigating risks from advanced AI. His concerns align with a growing field of AI safety research, which examines how to align machine goals with human values and prevent unintended consequences.

The debate over AI rights intersects with rapid technological progress. Tools like chatbots are increasingly integrated into daily life, from customer service to personal assistants, amplifying questions about their status and limits.

Bengio's call for robust guardrails emphasizes proactive measures. 'As their capabilities and degree of agency grow, we need to make sure we can rely on technical and societal guardrails to control them,' he said.

This position highlights the tension between innovation and safety in AI development, with implications for policymakers, ethicists and technologists worldwide.