Mythos and the Adolescence of AI Policy

Anthropic's unreleased AI model, Mythos, has triggered a series of events that are dramatically transforming the global AI policy and governance landscape | Edition #297

“The Shipwreck” by Joseph Vernet, 1772 (oil on canvas, modified)

Recent developments in AI have caused a seismic shift in the AI policy debate.

As someone who has been observing, documenting, and analyzing the AI policy and regulation zeitgeist daily for the past three and a half years, I can say that we have officially entered a new phase.

I will call it the adolescence of AI policy.

-

The first AI development that led to this shift, and perhaps the most consequential one, was Anthropic’s announcement of Mythos in early April.

The general-purpose AI model, which the company decided not to release to the public, is particularly capable at computer security tasks.

Anthropic wrote that during testing, Mythos Preview could identify and exploit vulnerabilities in every major operating system and every major web browser when directed to do so, including finding a 27-year-old bug in an operating system known primarily for its security.

Following up on the announcement of the unreleased model, Anthropic initiated Project Glasswing, a partnership with 50 initial partners, including companies, NGOs, labs, and the United States government, to strengthen cybersecurity and defensive capabilities.

According to the company, with the help of the model, these partners found over 10,000 high or critical-severity security flaws.

Five days ago, Anthropic announced it would expand Project Glasswing, allowing over 150 organizations in 15 countries to access the model.

One of the selection criteria for the partners was the provision of infrastructure services. If they were targeted by a cyberattack, millions of people could be affected, with potential ramifications for global and national security.

According to Anthropic, Mythos has already transformed the current cybersecurity paradigm and the conversations taking place within the cybersecurity field.

It assesses that other AI companies will have Mythos-class models within 6 to 12 months. These could become publicly available without safeguards to prevent misuse.

In practice, this means that cyberattacks, including those affecting critical infrastructure and impacting millions of people simultaneously, could become common, and sooner than many expect.

The significance of this development cannot be overstated: Anthropic’s concern is the textbook description of systemic harm or AI getting out of control. And it is no longer a theory or a mathematical model based on scaling laws; it is a risk directly associated with a model that exists today.

A cascade of AI governance implications has followed the announcement of Mythos and continues to unfold.

As we know, adolescence is a turbulent period in life, and global AI policy is now entering this turbulence.

-

Mythos did not go unnoticed by the world’s governments, militaries, or even the Vatican.

A month after Mythos’s release, Pope Leo XIV officially announced his new Encyclical called Magnifica Humanitas (or Magnificent Humanity).

I do not think that it is a coincidence that Anthropic’s co-founder, Chris Olah, was the only member of the AI industry invited to speak at the Encyclical’s presentation on May 25.

The document was widely read and discussed in AI governance circles, showing that, despite coming from a religious figure, its pro-regulation, pro-human stance aligned with the mainstream zeitgeist.

Among many other topics, the Encyclical discusses the need for diplomacy, multilateralism, regulation, and an economy that values dignity. It also mentions the rise of AI-powered cyberattacks and some of the ethical issues of the military use of AI.

It is almost surprising that the Pope chose to go into such detail on AI policy challenges and recommendations.

It is likely that the Vatican sensed the turbulent adolescence of AI policy coming and realized that, if it wanted to remain a globally influential institution, it was time to make its stance clear.

-

The third and fourth developments I want to comment on illustrate, in a more practical sense, what the adolescence of AI policy looks like.

These events took place in the past few days in the United States, and they are a direct reaction to the release of Mythos in April.

On June 2, President Trump signed a new Executive Order on AI, titled “Promoting Advanced Artificial Intelligence Innovation and Security.”

One of the focuses of this EO was to promote a voluntary framework for frontier AI developers to engage with the government before releasing “covered frontier models” in order to help protect the United States’ critical infrastructure, cyber defense capabilities, and national security, mainly against external attacks.

The trigger and the blueprint for this Executive Order were likely Mythos and Anthropic’s Project Glasswing (both of which I wrote about above).

The White House was likely very impressed by Mythos’ capabilities and decided that Anthropic’s approach was an advisable framework to implement for future highly capable AI models.

Contrary to what many media outlets and commentators wrote at the time, the White House is not seeking a mandatory registration system or some sort of vetting scheme before models can be launched, as it is explicitly clarified in the Executive Order itself:

“Nothing in this section shall be construed to authorize the creation of a mandatory governmental licensing, preclearance, or permitting requirement for the development, publication, release, or distribution of new AI models, including frontier models.”

This would also be against America’s AI Action Plan and the current administration’s approach to AI governance.

However, many were surprised by this Executive Order, as it seemed to stand out from President Trump’s general aversion to regulation and oversight.

The administration likely assessed that Mythos-like capabilities were unprecedented and required a different course of action.

Three days later, the White House released yet another document: a memorandum to accelerate the use of AI by the U.S. military.

This is also an indirect reaction to Mythos, but from the government’s military wing: if we are reaching new stakes in AI model capability, the United States’ military must be “all in” and quickly catch up.

This new Memorandum is also a direct response to the recent dispute between the U.S. Department of War and Anthropic over the company’s refusal to waive its own redlines on permitted and prohibited uses of AI by the U.S. military, which I analyzed here.

If you remember, the Department of War wanted to use Anthropic’s models whenever federal law allows, but Anthropic wanted a firm guarantee that its redlines banning AI for mass domestic surveillance and the use of fully autonomous weapons would be respected, regardless of what the law says, as the law might not be adequate to address emerging AI challenges.

The conflict led to Anthropic’s designation as a supply chain risk. A few days later, Anthropic filed a lawsuit against the U.S. federal government contesting this designation.

Here, in the June 5 Memorandum, we see the White House responding further to that incident and doubling down.

It establishes various measures to accelerate AI adoption by the U.S. military, explicitly stating that no company should interfere with how the military uses AI, including by disabling, degrading, or modifying its AI models, without the federal administration’s approval.

Therefore, Mythos prompted a twofold reaction from the White House: increased awareness of risks and the need for tighter oversight and stronger cybersecurity capabilities, and, simultaneously, an urge to accelerate the military’s broad adoption of new AI capabilities independent of AI developers’ internal redlines.

-

The AI policy shift I am describing is directly correlated with signs of a temperature increase in the AI risk thermometer and a general sense of a potential, but unforeseeable, loss of control.

Last week, the MIT AI Risk Initiative released another report that revealed how 272 experts assessed the severity of AI risks across various sectors, as well as how to mitigate them.

Not so optimistically, the report states that even if pragmatic mitigations were implemented, these experts judged five risks to have a greater than 10% probability of catastrophic outcomes: dangerous capabilities, weapons and cyberattacks, environmental harm, inequality and unemployment, and power centralization.

Still on the topic of rising risk, another recent development has further contributed to this sense of an imminent loss of control: signs that recursive self-improvement (RSI) might be approaching.

According to an article published three days ago by Anthropic, the company outlined what could happen if AI systems become capable of full recursive self-improvement and begin building their successors:

“In this world, the pace of progress in AI development becomes determined entirely by the availability of compute (or the speed of discovering various efficiencies in algorithmic training or inference) for AI systems. Humans play a substantially diminished role in their development, likely moving most of our effort towards oversight, validation, and verification of an expanding ‘virtual lab’ run by AI systems. We expect that systems capable of automated AI research and development would have skills that would transfer to the rest of science, allowing them to begin to revolutionize other fields.”

And it continues into more dystopian territory:

“How the alignment problem gets solved—or not—in this future is something we are least certain about. Models could prove to be sufficiently aligned and capable enough of research taste that they discover and implement novel solutions that we have not yet reached. They could also be sufficiently wise to halt development if not. Alternatively, the rare occurrences of misalignment present in today’s models could compound as the models build their successors, growing more frequent but less understood until we lose control of them. It’s possible that we can’t build, integrate, and verify the tools that we’d need to understand which trendline we are actually on.”

In its recent blueprint for a federal AI governance framework, OpenAI has also cited the risk of recursive self-improvement as a reason to ensure that AI development serves human interests:

“At the same time, increasingly capable AI systems are beginning to demonstrate abilities that raise concerns about cyber offense, biological misuse, autonomy, alignment, and other threats to national security.”

A few months ago, many countries, including the United States, were following the “wait and see” AI policy playbook. It would gain them time to grow their AI industry and avoid stifling innovation.

In a matter of weeks, Mythos, a model with extreme cybersecurity (and cyberattack) capabilities, was announced; a cyberdefense partnership (Glasswing) was formed; and two major AI companies are now warning about the consequences of AI building its successors alone.

Passive playbooks will no longer work because AI policy has entered its adolescence. It is time to transition to something else.

-

If I were to summarize the adolescence of AI policy, I would say that, for now, it is characterized by a series of uncoordinated, reactive, erratic, and aggressive measures in response to a sharp, accelerated, and uncontrollable (but predictable) rise in AI capabilities and risks.

As with Dario Amodei's Adolescence of Technology, this might also be simply a rite of passage. A short, stormy path that will prepare us for a more stable, mature stage.

But it remains unclear.

I will keep you posted.