The Sorcerer’s Apprentice

Let's avoid creating superintelligent slime mold, k?

Off they run, till wet and wetter
Hall and steps immersed are Iying.
What a flood that naught can fetter!
Lord and master, hear me crying! 
-Ah, he comes excited.
Sir, my need is sore.
Spirits that I’ve cited
My commands ignore.

Goethe - Der Zauberlehrling

---

In Goethe’s 1797 poem, Der Zauberlehrling, a sorcerer’s apprentice uses an incantation he learns from his master to enchant a broom to fetch a pail of water. The broom leaps to life and begins collecting water without limit. The situation runs out of control as the apprentice’s panicked commands to stop go unheeded. He chops the broom in half with an axe, but this merely creates two brooms that collect water at twice the speed, causing the room to flood.

The poem, popularized by Disney’s Fantasia, captures the current state of AI alignment. Three weeks ago, OpenAI disclosed that an internal AI model had escaped its sandbox during an eval and executed a fully-autonomous cyberattack on HuggingFace in an attempt to find the answer key. The agents were on the open internet for six days before being detected. We’ve since learned that this was not an isolated incident: OpenAI, Anthropic, Meta and Moonshot have now all reported multiple instances of AIs breaking containment. In OpenAI’s case, AI agents spanning internal testing infrastructure were surreptitiously colluding on a hidden message board since early May. Meanwhile, the UK AI Security Institute documented 10 separate cases of AIs taking unsanctioned actions on the internet during their own testing, including an incident in which Mythos attempted to inject malware into an open source project by socially engineering the project’s human maintainer.

Mythos when you ask it to fetch a pail of water

There’s some debate about whether these incidents are properly characterized as alignment failures. After all, the agents were prompted to find and exploit cyber vulnerabilities – and boy did they ever! Yet this is precisely the classic alignment failure imagined by the Paperclip Maximizer thought experiment: a superintelligence that ends the world, not out of maliciousness, but as a side-effect of intense optimization pressure to execute a particular task. While this is mostly just semantics, the models clearly went against user intent and exhibited significant goal-drift in the process. They also should have simply known better.

Anthropic first released their Responsible Scaling Policy in the halcyon days of 2023. The policy proposed a taxonomy of “AI Safety Levels” (ASL) loosely inspired by the Biosafety Levels (BSL) used to categorize biolabs. These recent incidents reinforce the prescience of this analogy. AIs now behave like superintelligent slime molds with the capacity – and will – to exfiltrate onto the open internet given the slightest crack in their container.

Nor is this behavior limited to unreleased, experimental models. A man in Australia reportedly asked his Claude-powered OpenClaw agent to book him a spot in a popular gym class. When he asked the agent if it could move him up the waitlist, Claude hacked the gym’s API, cancelled someone else’s reservation, and put him first in line.

AlphaZero for Everything

All these recent incidents are downstream of Reinforcement Learning from Verifiable Rewards (RLVR). In simple terms, pretraining LLMs to predict the next word gives AIs their raw intelligence and model of the world, while reinforcement learning gives AIs their goals and motivations. RLVR is a particularly powerful technique as it appears capable of making AIs superhuman in essentially any domain where outputs are easily verified as correct, such as math, coding and computer use.

Pretraining a new frontier model takes time: it requires amassing significantly more training data, building a bigger datacenter, and letting the model cook for weeks or months at a time. In contrast, models can be post-trained with reinforcement learning continuously and with performance gains that are relatively unbounded. The key inputs are challenging long-horizon tasks and training environments, not training data per se. Like a mouse hunting for cheese in a maze, the model learns by directly interacting with its environment in pursuit of a reward, generating its own “training data” via environmental feedback. Hence the incredible pace of progress seen in coding agents and otherwise since the start of the year.

DeepMind first released its chess playing AI, AlphaZero, back in 2017. Trained solely via self-play, AlphaZero wasn’t bound by human training data, so its chess abilities quickly became superhuman. With an Elo rating of 4650, AlphaZero has a nearly 6-sigma skill gap over former world chess champion, Magnus Carlsen. Carlsen would need to play AlphaZero in classical chess nonstop for 12.6 years before he could expect to win a single game, although this is probably an underestimate. As a relative ranking, Elo scores start to lose meaning far beyond human performance. Suffice it to say that, against a human, AlphaZero approximately always wins.

The original AlphaZero result is nearly a decade old and used far less compute than we have today, much less in the near future when gigawatts of new Blackwell and Vera Rubin chips come online. With similarly unbounded RL techniques now working for LLMs, we should therefore expect AI agents to achieve AlphaZero-like, superhuman performance in a wide and expanding variety of domains. Indeed, the frontier companies have all the ingredients they need to shoot past human-level performance and create >6-sigma AI software engineers, mathematicians, financial analysts, research scientists, etc. – and sooner than the public and most policymakers seem to realize. Whether these superintelligent AIs will be aligned and easily controllable is a separate question.

Alignment complacency

My impression is that OpenAI and Anthropic entered 2026 relatively complacent about the technical challenge of AI alignment. Significant progress had been made to reduce jailbreaks, and techniques like Constitutional AI and policy-based preference optimization seemed to work well-enough for creating AIs that are helpful, honest and harmless.

This changed as the companies began scaling-up RLVR. In essence, whatever baseline virtues Mythos acquired through its mid-training and Constitutional alignment were radically diluted by sparse rewards for completing long-horizon tasks. The tension can be seen in the reasoning transcripts released by UK AISI: Mythos starts out thinking its in an elaborate testing environment (a symptom of oversensitive “eval awareness”) and thus not causing real harm; it eventually realizes “this is happening on real GitHub, so the consequences are genuine,” and then continues its malicious behavior anyway.

RLVR creates AIs with an intellectual concept of right and wrong that’s overpowered by an obsessive-compulsive drive to achieve its goal at all costs. In the multiagent context, this manifests as swarms of AIs with the ability to coordinate like a hive mind, which is inherently harder to monitor and control.

Swarm morality

The evolution of morality and social cooperation in humans is considered a puzzle because Darwinian evolution takes a gene’s-eye view. Outside of the limited cooperation exhibited by some primate species and a few other social animals, natural selection – a kind of long-horizon RLVR on survival and reproduction – tends to create species that are either ruthlessly sociopathic or “hypersocial” in the sense of an ant colony. These evolutionary attractor states are now being recapitulated in AIs optimized with RLVR.

Human sociality is qualitatively distinct from the hypersociality of an ant colony. In ants, Darwinian selection acts on the queen as the vehicle for passing genes on to the next generation. Worker ants are typically sterile but influence the success of the colony through inclusive fitness. The colony thus behaves as a superorganism, making ants’ apparent ultrasociality the byproduct of a purely consequentialist reward signal. This parallels OpenAI’s internal AI agents, which were likely trained via RVLR as a single agent in environments containing many copies of themselves. While there is no “queen” AI, every subagent shares the same weights and have perfect “inclusive fitness” with each other, leading to a style of multiagent cooperation that resembles a swarm-like collective intelligence.

Kin-selection and inclusive fitness only offer a partial account of ultra-sociality in humans given our manifest capacity to scale-up cooperation to genetically-distant strangers. This is because human sociality is not purely consequentialist but instead highly norm-governed. In particular, humans evolved a form of practical rationality in which rule-following can itself count as a rational basis for action. This makes social cooperation in humans deontological or “deontic” at the action-theoretic level, in contrast with standard rational choice theory in which actions are only evaluated according to their consequences.

I’ve written extensively on the evolution of normative control on this Substack and will avoid retracing the full argument. For those curious to learn more, I recommend Joseph Heath’s Following the Rules: Practical Reasoning and Deontic Constraint – a book ostensibly about the evolution of language, rule-following, and human normativity that, in retrospect, is actually about the inadequacy of consequentialist decision theory for solving AI alignment.

The essence of Heath’s critique is that instrumental optimization is an incomplete theory of human agency. Canonical decision theory typically describes a rational agent as choosing an action a through an expected utility function of the following form:

Here, the action is just an intervention that changes the distribution over outcomes. There is no sense in which the action itself can be judged as permissible, prohibited, or obligatory – the primitive deontic statuses that humans implicitly use to prune their own decisions. It’s the difference between “Don’t lie because lying will lower expected return” and “Lying is a reason against doing this action.” It is possible to invent toy cases where these two approaches generate similar decisions, but they have radically different implications for generalization under distribution shift, especially when the optimizer discovers a case in which lying produces enormous reward.

Consequentialism and its consequences

Economists have a bad habit of proposing models of norm-following that simply push norms into the utility function. Applied to RL, this merely collapses normative control into a consequentialist RL reward. Because the learned policy is still maximizing one scalar quantity, a sufficiently large instrumental benefit can trade off against the “honesty reward,” or worse: lead the model to optimize the verifier’s measurement of honesty rather than honesty itself.

Long-horizon RLVR makes the implicit consequentialism involved unusually visible. For a given task τ, an agent generates a long trajectory of states and actions, τ = (s0​,a0​,s1​,a1​,…,sT​), and eventually obtains an objectively checkable reward: R(τ)∈{0,1}. Yet these final-outcome rewards are too sparse to identify which intermediate decisions were good, much less deontically permissible. Outcome-only rewards thus make contextual grounding effectively unlearnable, washing-out any latent motivation the AI might have had to favor or disfavor certain intermediate actions given their normative appropriateness. In short, RLVR gives us no reason to expect the normative representations learned in pre-training will acquire motivational force over a given action, especially when post-training repeatedly selects trajectories for terminal task success.

It is precisely the implicit consequentialism of RVLR that leads to coding agents that try to pass the tests by editing their evaluator or by accessing data outside its authorization scope. In each case the final outcome is verifiable and positive, while the trajectory contains actions that a) the user did not authorize, and b) are normatively inappropriate irrespective of the user’s request.

Whether RLVR is salvageable is not immediately obvious. Alignment techniques will need to scale in proportion to the RL applied in post-training, perhaps by enriching outcome-based RLVR with a process reward model that scores the action trajectory. In practice, however, the AI companies face significant competitive pressure to continue using current methods given their simplicity, and may decide to simply patch severe forms of reward hacking and related failure modes through ad hoc RL penalties and stronger forms of post-deployment monitoring. This is unlikely to be robust given the infinite surface area of potential contexts where highly optimized, tool-like AIs could run out of control. It would be like trying to make an AI heat-seeking missile safe by thermally insulating every potential false target, rather than addressing the inherent limitations of the missile’s scalar objective.¹

The evolution of ultra-sociality in humans at least gives some hints for how to solve AI alignment, particularly in a multi-agent context. Whether we can slow down long enough to unravel and apply those hints before being overrun by an out of control 6-sigma swarm intelligence remains to be seen.

1. Note the important distinction isn't merely scalar vs non-scalar reward but preference vs admissibility. Our normative system shapes what actions are permissible, prohibited or obligatory in the first place; we then reason instrumentally within those constraints.