"Conscious AI" as an AI Safety Issue
Due to emerging AI risks and their legal and ethical implications, misleading, exaggerated, or false claims of "conscious AI" should be treated as an AI safety issue | Edition #290
Hi everyone, this is Luiza Jarovsky, PhD. Welcome to the 290th edition of my newsletter, trusted by 94,400+ subscribers worldwide.
As AI disrupts the knowledge economy, this is a great time to differentiate yourself and take the lead. Here is how my AI, Tech & Privacy Academy can help you:
Join the 29th cohort of my AI Governance Training (3 seats left). You can register with a group discount or apply for an individual discount.
Sign up for educational resources at our Learning Center
Discover your next great read in my AI Book Club
Listen to my AI governance talks with global experts
Explore leading papers at our AI Ethics Paper Club
Receive our Job Alerts with AI governance openings
Check out our sponsor: AI Forensic Agents
In the 1800s, steam explosions killed thousands before regulators enforced accountability. Today, AI systems can cause harm at scale, including data leaks and security flaws, while responsibility is diffused. AI Forensic Agents identifies root causes and assigns responsibility. Download the paper.
“Conscious AI” as an AI Safety Issue
I have been writing about AI governance for over three years, and I have noticed that the public's attitude toward the possibility of AI consciousness (and the implications that this possibility entails) has changed dramatically in recent years.
Increasingly, I see people expressing opinions that reflect a belief that AI has consciousness, emotions, morals, and ideas of its own.
In this view, AI is seen as some sort of self-aware entity now populating the world, a type of alien intelligence, which should be respected, integrated, embraced, and even idolized.
If you do not believe me, or have not noticed this trend, I invite you to read the comments on my recent post about this paper on seemingly conscious AI risks, which was viewed by over 125,000 people on X:
Every time I discuss AI companies’ practices that unduly encourage this kind of AI hype (such as when I criticized Anthropic's new “constitution” for Claude for irresponsibly fostering AI anthropomorphism and legally questionable theories of AI personality), I receive personal insults from people who are convinced that their AI systems are conscious entities.
The spread of the “conscious AI” myth
Large language models are trained on human text scraped from the internet, social media, books, and other human sources, and are fine-tuned to be conversational, direct, personalized, friendly, agreeable, and often sycophantic.
Many people who interact with LLM-powered AI systems end up becoming dependent on, emotionally attached to, and even romantically involved with them, as I have covered in this newsletter over the past few years.
Many people report feeling that some sort of independent and conscious entity has emerged.
We do not have enough data to determine exactly why people start to feel that way and project consciousness onto AI systems, but we can hypothesize.
Throughout the thousands of years that the human brain has been evolving, it has always associated the use of human language with the formation of human relationships, bonds, and strategic partnerships that would enable procreation and survival.
When there were language-based two-way interactions, there were other humans, which is likely hard-coded somewhere in our brains.
Since late 2022, for the first time in history, hundreds of millions of humans have been using human language to interact bidirectionally with machines.
As the spread of the “conscious AI” myth shows, this has confused many people who continue to associate language with human interaction and project consciousness onto non-sentient machines.
However, this is not the only reason the myth of “conscious AI” has spread dramatically over the past few months.
Many influential voices in the AI industry have taken advantage of the complexity and challenges of theoretical discussions about consciousness and have embraced a particularly broad functionalist approach to it.
According to functionalism in the philosophy of mind:
“what makes something a mental state of a particular type does not depend on its internal constitution, but rather on the way it functions, or the role it plays, in the system of which it is a part.”
Applying this approach to a potential “AI consciousness,” AI systems could be considered sentient or conscious based on computational scale or increasing algorithmic complexity, as a recent paper, “The Abstraction Fallacy,” criticized.
The best criticism I have read so far of the functionalist view of AI consciousness was written by neuroscientist Anil Seth earlier this year, in his article “The Mythology Of Conscious AI.” He wrote:
“A computational simulation of the brain (and body), however detailed it may be, will only give rise to consciousness if consciousness is a matter of computation. (…) This brings us back to the poverty of the brain-as-computer metaphor. If you think that everything that matters about brains can be captured by abstract neural networks, then it’s natural to think that simulating the brain on a digital computer will instantiate all its properties, including consciousness, since in this case, everything that matters is, by assumption, algorithmic.”
New forms of AI will likely emerge in the coming years, and they might be capable of building deeply complex and intelligent simulations of the Universe, the human mind, and everything that can possibly be known, felt, expressed, or understood by humans.
However, from an AI governance perspective, we need to be able to separate what we call human consciousness, which is inherently tied to our human biology (at least the way we experience it), from whatever other simulations or machine-awareness that might be observed or manifested.
This distinction is necessary due to the risks associated with attributing consciousness, sentience, and moral patienthood to AI, which are largely unknown to the general public: