👁️ Can Humans Really Oversee AI?
Emerging AI Governance Challenges | Paid Subscriber Edition | #175
👋 Hi, Luiza Jarovsky here. Welcome to our 175th edition, read by 54,200+ subscribers in 165+ countries. Not a subscriber yet? Join us.
🌎 We are a leading AI governance publication helping to shape the future of AI policy, compliance, and regulation. It's great to have you here!
👁️ Can Humans Really Oversee AI?
Dozens of countries have already enacted or are currently debating their AI laws. Various drafts and approved AI laws, including the EU AI Act, rely on human oversight requirements as a way to mitigate risks and prevent AI-related harm.
In this edition of the newsletter, I argue that there is an overreliance on human oversight in legal frameworks, and many conveniently avoid discussing its shortcomings. In many cases, human oversight is extremely difficult to implement, and I want to share why.
1️⃣ What the EU AI Act Says About Human Oversight
I'll start with the world's most famous AI law: the EU AI Act.
Article 14 of the AI Act establishes human oversight as one of the requirements for high-risk AI systems:
“1. High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use.”
Every AI system classified as high-risk under the EU AI Act must comply with the human oversight requirement throughout its entire period of use.
Additionally, the EU lawmakers used the term “effectively overseen,” meaning it should not be merely a formalistic measure but a genuine way to prevent or minimize risks.
“2. Human oversight shall aim to prevent or minimise the risks to health, safety or fundamental rights that may emerge when a high-risk AI system is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, in particular where such risks persist despite the application of other requirements set out in this Section.”
The second paragraph clarifies that the goal of human oversight measures is to protect people and prevent or minimize risks to fundamental rights.
An important element here is that the oversight measure should be compatible with the proper use of the AI system, meaning its use according to its intended purpose and in conditions of reasonably foreseeable misuse.
“3. The oversight measures shall be commensurate with the risks, level of autonomy and context of use of the high-risk AI system, and shall be ensured through either one or both of the following types of measures:
(a) measures identified and built, when technically feasible, into the high-risk AI system by the provider before it is placed on the market or put into service;
(b) measures identified by the provider before placing the high-risk AI system on the market or putting it into service and that are appropriate to be implemented by the deployer.”
The third paragraph clarifies that the oversight measure should not be a one-size-fits-all solution and that it can be either (or both) built-in by the AI system's provider or implemented by its deployer.
The fourth paragraph of this article details how human oversight should ideally work in practice:
“4. For the purpose of implementing paragraphs 1, 2 and 3, the high-risk AI system shall be provided to the deployer in such a way that natural persons to whom human oversight is assigned are enabled, as appropriate and proportionate:
(a) to properly understand the relevant capacities and limitations of the high-risk AI system and be able to duly monitor its operation, including in view of detecting and addressing anomalies, dysfunctions and unexpected performance;
(b) to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons;
(c) to correctly interpret the high-risk AI system’s output, taking into account, for example, the interpretation tools and methods available;
(d) to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output of the high-risk AI system;
(e) to intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state.”
This paragraph states that human oversight measures should allow:
understanding of the capabilities and limitations of the AI system;
monitoring the AI system;
awareness of automation bias;
interpreting the output;
deciding to ignore, reverse, or interrupt the AI system when necessary.
Well done to the EU lawmakers for being extremely detailed in this article, providing a baseline for other countries currently considering their AI governance and regulation mechanisms.
However, in many cases, implementing effective human oversight measures—especially those that follow detailed specifications like those in the EU AI Act—will be extremely challenging. There are two main reasons for this:
2️⃣ The Black Box Paradox
AI systems, especially those based on machine learning, are trained on vast amounts of data and are designed and continuously fine-tuned to operate with varying levels of autonomy, identifying patterns within the data they process.
Often, and depending on the context in which they are deployed, it can be difficult to understand why an AI system arrived at a particular solution. We might find the outcome persuasive or reasonable, but we may not be able to explain why the AI system chose outcome A instead of outcome B.
When dealing with high-risk AI systems under the EU AI Act—such as those used in employment, education, public services, and law enforcement—the black box paradox becomes a serious concern. These AI systems may curtail freedoms or directly impact access to fundamental services, so the reasoning behind each decision should be clear, explainable, justifiable, and contestable.
According to the EU AI Act, the human responsible for oversight measures must be able to understand how the AI system operates and interpret its outputs, intervening when necessary to prevent harm to fundamental rights.
But if AI systems are highly complex and function like black box—operating in an opaque manner—how are humans supposed to have a detailed comprehension of their functioning and reasoning to oversee them properly?
If we accept that humans often won't fully grasp an AI system's decision-making, can they decide whether harm to fundamental rights has occurred? And if not, can human oversight truly be effective?
3️⃣ Automation Bias
Automation bias is another important factor to consider when trying to understand the effectiveness of human oversight measures.