🤺 AI Battles: Oversight & Explainability
AI Governance Professional Edition | Paid-Subscriber Only | #151
👋 Hi, Luiza Jarovsky here. Welcome to the 151st edition of this newsletter on AI policy, compliance & regulation, read by 40,500+ subscribers in 155+ countries. 🙏 Happy Thanksgiving to those celebrating!
🔒 This is an exclusive AI Governance Professional Edition featuring my critical analyses of AI compliance and regulation topics to help you stay ahead in the fast-paced field of AI governance. It’s highly recommended for legal, privacy, and tech professionals, as well as anyone seeking to stand out in AI governance. If you're not a paid subscriber yet, upgrade here.
⏰ Only 4 days left! This December, join me for a 3-week intensive AI Governance Training program (8 live lessons; 12 hours total), already in its 15th cohort. Over 1,000 people have benefited from our programs: learn more and register here. Use coupon code BLFRI2024 to enjoy a special 20% off Black Friday discount, valid only until Monday!
🤺 AI Battles: Oversight & Explainability
This week, I want to discuss two central AI ethics principles - human oversight and explainability—and highlight some of the practical challenges when translating them into legal provisions.
1️⃣ AI Ethics Principles
➜ OECD AI principles
The OECD AI principles—updated in May 2024—are considered central pillars to guide lawmakers, policymakers, and AI governance professionals in their AI policy and regulation efforts.
Regarding human oversight, OECD AI principle 1.2 states:
“AI actors should respect the rule of law, human rights, democratic and human-centred values throughout the AI system lifecycle. These include non-discrimination and equality, freedom, dignity, autonomy of individuals, privacy and data protection, diversity, fairness, social justice, and internationally recognised labour rights. This also includes addressing misinformation and disinformation amplified by AI, while respecting freedom of expression and other rights and freedoms protected by applicable international law.
To this end, AI actors should implement mechanisms and safeguards, such as capacity for human agency and oversight, including to address risks arising from uses outside of intended purpose, intentional misuse, or unintentional misuse in a manner appropriate to the context and consistent with the state of the art.”
Regarding explainability, principles 1.3 states:
“AI Actors should commit to transparency and responsible disclosure regarding AI systems. To this end, they should provide meaningful information, appropriate to the context, and consistent with the state of art:
- to foster a general understanding of AI systems, including their capabilities and limitations,
- to make stakeholders aware of their interactions with AI systems, including in the workplace,
- where feasible and useful, to provide plain and easy-to-understand information on the sources of data/input, factors, processes and/or logic that led to the prediction, content, recommendation or decision, to enable those affected by an AI system to understand the output, and,
- to provide information that enable those adversely affected by an AI system to challenge its output.”
➜ EU Ethics Guidelines for Trustworthy AI
From an EU perspective, in 2019, the EU High-Level Expert Group on AI presented the Ethics Guidelines for Trustworthy AI, establishing seven key requirements that AI systems should meet to be considered trustworthy.
Regarding human oversight, the first key requirement establishes:
“Human oversight helps ensuring that an AI system does not undermine human autonomy or causes other adverse effects. Oversight may be achieved through governance mechanisms such as a human-in-theloop (HITL), human-on-the-loop (HOTL), or human-in-command (HIC) approach. (…). Oversight mechanisms can be required in varying degrees to support other safety and control measures, depending on the AI system’s application area and potential risk. All other things being equal, the less oversight a human can exercise over an AI system, the more extensive testing and stricter governance is required.”
Regarding explainability, the fourth key requirement states:
“(…). Technical explainability requires that the decisions made by an AI system can be understood and traced by human beings. Moreover, trade-offs might have to be made between enhancing a system's explainability (which may reduce its accuracy) or increasing its accuracy (at the cost of explainability). Whenever an AI system has a significant impact on people’s lives, it should be possible to demand a suitable explanation of the AI system’s decision-making process. Such explanation should be timely and adapted to the expertise of the stakeholder concerned (e.g. layperson, regulator or researcher). In addition, explanations of the degree to which an AI system influences and shapes the organisational decision-making process, design choices of the system, and the rationale for deploying it, should be available (hence ensuring business model transparency).”
2️⃣ Ethics vs. Law
Both human oversight and explainability have been the topic of numerous social science research papers and policy reports, and there is no doubt that they are core ethical principles that should be guiding AI development and deployment, as well as AI lawmaking and policymaking.
The main challenge at this point is regulatory: translating these ethical principles into legal provisions. Law often establishes principles that will guide the interpretation and enforcement of the provisions in a certain field. This is a fundamental aspect of legal theory and practice, with the content and methods differing based on the jurisdiction.
However, in some instances, we have ethical imperatives or key requirements to implement ethical goals (using the EU High-Level Expert Group on AI's terminology). In such cases, the law must go beyond the statement of the principles and establish concrete provisions and obligations to guide behavior in the desired direction.
In my view, this is where we currently stand in the field of AI law and governance. While so many countries are currently focused on enacting and enforcing their AI laws, we should make sure that decades of discussion around AI ethics are properly translated into effective legal provisions and obligations.
However, this is not an easy task, and many are underestimating the “AI battles” ahead—instances where humans will be “battling” AI systems to ensure compliance with legal requirements.
Let's use the EU AI Act as a case study and look at some of the existing challenges regarding explainability and human oversight, both from the perspective of companies (attempting to comply) as well as from the perspective of EU authorities (enforcing the provisions).
3️⃣ Explainability and Human Oversight Challenges in the EU AI Act
➜ Explainability
Recital 27 of the EU AI Act highlights that the 2019 Ethics guidelines for trustworthy AI presented by the EU High-Level Expert Group on AI (see item 1 above) apply to the AI Act in the context of “non-binding ethical principles for AI, which are intended to help ensure that AI is trustworthy and ethically sound.”
Specifically on explainability, Recital 27 states:
“(…) Transparency means that AI systems are developed and used in a way that allows appropriate traceability and explainability, while making humans aware that they communicate or interact with an AI system, as well as duly informing deployers of the capabilities and limitations of that AI system and affected persons about their rights. (…)”
Recital 59 highlights that explainability is a tool to help ensure that procedural fundamental rights are exercised, such as the right to an effective remedy, to a fair trial, as well as the right of defense and the presumption of innocence:
“Furthermore, the exercise of important procedural fundamental rights, such as the right to an effective remedy and to a fair trial as well as the right of defence and the presumption of innocence, could be hampered, in particular, where such AI systems are not sufficiently transparent, explainable and documented.”
The main article in the EU AI Act covering explainability—or the right to explanation—is article 86. It states:
“1. Any affected person subject to a decision which is taken by the deployer on the basis of the output from a high-risk AI system listed in Annex III, with the exception of systems listed under point 2 thereof, and which produces legal effects or similarly significantly affects that person in a way that they consider to have an adverse impact on their health, safety or fundamental rights shall have the right to obtain from the deployer clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken.”
╰┈➤ Practical questions regarding the enforcement of this provision:
- Having in mind the black box paradox, or the inability of AI users, or even its developers, to explain a decision taken by an AI system, how will AI deployers comply with this obligation?
- How will AI deployers explain in a clear and meaningful way the role of the AI system in the decision-making procedure? Is it technologically possible?