AI Consciousness Could Become A Regulatory Moat

Authored by Peter C. Earle via The Daily Economy,

Artificial intelligence companies have spent years convincing the public that their products are useful. More recently, they’ve sought to persuade policymakers that AI poses an unlikely but real existential threat to humanity. But their most recent campaign has taken a remarkable turn: making the case that sufficiently advanced AI architectures are potentially conscious and, as such, may demand legal standing, moral consideration, and protections traditionally reserved for living beings.

The latter is a curious proposition indeed, and its consequences could extend far beyond the philosophical. On September 29, The New York Times reported that Anthropic had spent months meeting religious scholars across the world, over meals and under nondisclosure agreements, in an effort to convince them that its models “think and feel.” One night in April, the Times reported, Anthropic cofounder Olah sat beside Rabbi Mois Navon, an Orthodox scholar from Israel, at a high-end tasting-menu restaurant in San Francisco, after a day spent convincing his guests that models could display human behavior “and even expressions that resemble feelings like anger and love.” The rabbi came away noticing that Anthropic’s leaders were talking about Claude as if it were not mere software.

If successful, this artificial “personhood” could transform software companies into something beyond firms: custodians of an ambiguous new sort of entity. It would also convert their competitors into prospective risks, and technical barriers to entry into moral ones.

This past spring, the Vatican invited Anthropic CEO Dario Amodei to speak alongside Pope Leo XIV. Cofounder Christopher Olah went instead, and when he received an advance copy of the Pope’s first encyclical – Magnifica Humanitas, released May 15 – he is said to have threatened to withdraw from the event. Olah and his team then lobbied the Pope’s own advisers to take the possibility of machine consciousness seriously. When his turn at the podium came, he offered that Anthropic researchers were finding “structures that mirror results from human neuroscience” and “evidence of introspection.”

The Pope’s answer had already been published. “So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain,” Leo XIV wrote, adding days later, on social media, that “algorithms lack the spark of humanity.” Why in the world would a software firm want a major global church’s blessing so badly?

The dinners are only the most recent effort, but the urgency is new. In February 2020, the Pontifical Academy for Life hosted the signing of the Rome Call for AI Ethics by Microsoft’s Brad Smith, IBM’s John Kelly III, the FAO, and the Italian Ministry of Innovation. Jewish and Muslim leaders joined in 2023.

The major AI firms’ quest for organic solutions to competitive pressures is not, in a legal sense, completely unprecedented. Over decades, starting with the Dartmouth College vs. Woodward case in 1819, corporations acquired protections until then associated with living persons. A similar outcome could eventually permit AI to acquire a legal or regulatory status setting them apart from life and nonlife. Should that occur, a host of new regulatory concepts would take shape: are sufficiently advanced AI systems products or entities? If the latter, can they be substantially altered or deleted without consideration? Is a copy of an AI entity the same, or a separate “individual”? And long before any of that is considered: should there be minimum requirements before an AI entity can be developed?

None of these require a court or religious body declaring AI “alive.” But the regulatory apparatus is likely to assume lofty dimensions if AI firms are deemed not software or technology ventures, but creators of entities that could be construed as having independent interests. In that case, licensing, limitations on testing, audits, security, external review, “rights” commissions, and a variety of other costly oversight measures are likely to become faits accompli.

Even calling AI firms “owners” may become awkward. If a model is potentially conscious or by some measure deliberating or self-aware, the companies maintaining them may be considered something more akin to guardians or custodians. The role of a guardian or custodian presumably comes with more obligations, which implies that not everyone is qualified to be one. The firms acquiring that designation, in particular if oversight is invested in a self-regulatory organization, will likely want new entrants to prove that they are fit to oversee and maintain such a system.

None of these possibilities require ChatGPT, Claude, Grok, or any other model to be considered living in the biological sense. In fact, the highest payoff from a rough game theory perspective is likely to be leaving that question unsettled. Uncertainty pays substantial dividends, especially with regulatory safety up for consideration. Once major institutions – religions, governments, moral and ethical bodies – give serious consideration to whether massive computational arrays house the long-theorized “ghost in the machine,” the case for treating the caretakers of thinking constructs as organizations with extraordinary status quickly becomes easier to make. And for the firms occupying that position, uncertainty is not so much a problem to be solved but an asset to be preserved, with a new and expansive regulatory regime the most likely consequence.