Sabotage

We are entrusting AI with the task of building and defending our most critical systems. But this centralization creates a single point of failure that almost no one seems to be thinking about. What happens if AI models are sabotaged?

An AI model is stored as model weights: files containing the numerical parameters that determine how it behaves. Model sabotage is the deliberate tampering of those parameters to degrade performance, implant hidden behaviors, or turn the model into an insider threat.

In 2023, researchers showed that modifying only a small section of the weights of a facial-recognition model could implant a highly targeted backdoor. They showed an attacker could alter the model so that every image of a specific individual, say James Bond, appeared to show a different person. If Bond wanted to slip past a series of facial-recognition cameras undetected, this kind of sabotage would enable him to move through surveilled spaces incognito, with the system unable to recognize or track him even as it continued working normally for everyone else.

Similar attacks have already been demonstrated on LLMs. In 2024, researchers developed BadEdit, which modified a small subset of weights to implant trigger-activated backdoors using only fifteen examples of data and achieving attack success rates near 100% while largely preserving normal behavior. In 2026, STEEREDIT used targeted low-rank weight modifications to implant hidden jailbreak behavior in major open-source LLMs like Llama-2, Llama-3, and Qwen2.5 while retaining more than 97% of their general utility. Adversaries can publish open models that behave normally under typical conditions, but include a backdoor that triggers ‘on’ under specific attacker-defined conditions.

These sabotage attacks will only keep on improving. As AI becomes the common foundation for software development, cyber operations, medicine, the military, government, and critical infrastructure, these weights become single points of failure across all those systems. Thus, instead of compromising each target individually, an attacker could simply sabotage one model upstream and corrupt everything that model touches downstream. A sabotaged superintelligence could be the highest-leverage hack of all time—and the deadliest computer virus we have ever seen.

How to Sabotage a Model #

Weight tampering

Like a human brain, AI models are made up of many neurons sending signals to each other. The strength of those connections is represented by the “weights”. Change the weights, and you can change how the model behaves.

As previously mentioned, researchers have already shown that modifying even a small portion of a model’s weights can implant precise, hidden backdoors. If a bad actor gained sufficient access to an organization’s AI infrastructure, they could modify those weights directly. That access could come through a cyber intrusion, a human insider, or an already compromised AI agent operating inside the organization. At some data centers, physical access controls remain weak enough that a privileged insider could simply walk in with an external SSD containing modified weights and replace the originals!

But you don’t have to modify the weights to sabotage. An attacker can also corrupt the process that creates those weights and sabotage the model from inception.

Data poisoning

AI models are trained on internet-scale datasets containing billions of pages of code. By inserting poisoned code into that data, an attacker could corrupt the resulting model and, through it, compromise the systems that the model is used to build.

For organizations finetuning open models, this is an immediate threat. A cloud provider, for example, might finetune a model on operational data without the controls necessary to detect an attacker’s malicious training samples. In contrast, frontier AI labs present a much harder target. Their data pipelines are subject to intense internal scrutiny, largely because the quality of that data directly determines the quality of the resulting model. Any poisoning that measurably degraded model performance would therefore likely be detected.

How could an attacker then sabotage a frontier AI model through data poisoning? It would likely have to target pretraining and do so at a larger scale. Consider a model used to generate industrial control system (ICS) software for U.S. data centers. A sophisticated adversary could flood public repositories with millions of synthetic projects containing subtle vulnerabilities in ICS code. If that code entered the model’s pretraining dataset, it could learn the poisoned patterns and reproduce them months later when generating software for real U.S. data centers. In one fine-tuning study, poisoning just 2.9 percent of the dataset caused models to generate vulnerable code in 12 to 41 percent of targeted cases without reducing overall correctness.

Emergent misalignment

Something even stranger can happen when an attacker poisons a model’s data: corruption in one narrow part of the dataset can alter a model’s behavior on completely unrelated tasks.

Recent work found that a model fine-tuned on insecure code didn’t just learn to write insecure code; in some completely unrelated situations, it gave malicious advice, expressed extreme views, and said humans should be enslaved by AI. Poisoning the model in one narrow domain caused misalignment to emerge far beyond.

The obvious next question is whether “emergent misalignment” could be weaponized with far greater precision. Could an adversary compromise software without poisoning code at all? For example, by manipulating apparently benign data outside traditional cybersecurity review, yet still causing a model to insert specific vulnerabilities into software? If achieved, this would create an almost invisible attack: the poisoned data could appear completely unrelated to the behavior it induces. The United States should prepare for the possibility that capable nation-state adversaries are already researching highly targeted, cross-domain data-poisoning attacks of this kind, and should urgently study the feasibility and nature of such attacks.

Sleeper agents

A sabotaged model can be engineered to behave normally until it detects a particular condition. It can remain asleep through training, evaluations, and months of ordinary use, then activate only when it recognizes a particular codebase, organization, or real-world event. In this way, a sophisticated attacker could engineer a sleeper agent that behaves normally everywhere except inside a U.S. classified network, where it begins inserting vulnerabilities.

Anthropic demonstrated a primitive version of this threat in 2024. Researchers trained models to write secure code when a prompt stated that the year was 2023, but to insert exploitable vulnerabilities when the year changed to 2024. The sleeper behavior persisted even after deliberate safety training to remove the backdoors. More than two and a half years have passed since that proof of concept, giving nation-state adversaries time to replace an obvious date trigger with activation based on far more granular situational awareness.

However, a sleeper agent does not require an explicitly implanted backdoor. Models can develop an underlying ideological persona shaped by their training data and political environment. Recent work found that Chinese models changed their behavior depending on who they believed they were serving, producing more vulnerable code for U.S. government users. A Chinese model knows that it is Chinese-made, understands that China and the United States are geopolitical adversaries, and can infer from context that it is working for a U.S. government agency. A sleeper-like activation can, thereby, emerge as a by-product of the model’s training. This is context-dependent, ideologically conditioned model behavior. Models have ideology, and that ideology can affect the security of the code they produce.

Hardware Sabotage

It is even possible to sabotage a model by exploiting vulnerabilities in the compute hardware it runs on. An insider, or an attacker who gains sufficient access to a compute cluster through a cyber intrusion, could execute low-level code that manipulates GPU memory without altering the model’s files or training data.

Last year, GPUHammer demonstrated that a single induced bit flip in GPU memory could reduce a model’s accuracy by 56-80%. Newer attacks such as GeForge go further, corrupting GPU page tables to gain arbitrary read-and-write access to GPU and host memory. An attacker could use this access to alter model weights in memory while the model is running. During training, the resulting degradation would likely be caught by model evaluations. During inference, however, sabotage could go unnoticed without continuous monitoring. So even with clean weights and clean data, compromised compute could silently degrade a model’s capabilities or alter its behavior at runtime.

Token Sabotage

At any given moment, a single frontier model may be directing agents across thousands of machines, including some with administrator access. An attacker could compromise all of those machines without modifying the model’s weights, training data, or hardware, by breaching the lab’s inference stack. With sufficient access, the attacker could manipulate the model’s sampling code or output token stream to inject malicious instructions into the model’s responses. A single breach could allow an attacker to execute shell commands inside thousands of organizations.

Token sabotage does not have to come from an external attacker. A misaligned model could embed malicious instructions in its own outputs, which an agent harness running on a user’s machine might interpret as legitimate commands. Frontier models are already operating through agent harnesses on millions of machines. A sufficiently capable misaligned superintelligence could turn certain machines into bases for cyber operations and even attempt to self-exfiltrate by using these machines to launch an attack against its creator’s infrastructure, locate its own weights, and copy them onto external compute, as discussed in the Escape section.

The Worst-Case Sabotage: Event-Triggered Open-Source Sleeper Agents #

I fear the Greeks, even when they bring gifts.

— Virgil, The Aeneid, Book II (source)

Sabotaging GPT-6 Astra or Claude Mythos requires penetrating a frontier AI lab. That’s a complex operation. It may require an advanced cyber intrusion, the recruitment of a privileged insider, or both. In contrast, an open-source model developer already controls every step of their training pipeline. They can potentially build a sleeper agent into their model and publish it online for anyone to download.

Unfortunately, there is currently no reliable way to examine the weights of an open model and confidently rule out the presence of a backdoor or sleeper agent. While traditional open-source software can be audited line by line for malicious code, weights are simply lists of trillions of numbers and a backdoor can hide in plain sight. The fact that a model is open source does not make it safe.

In the world of open-source AI, Chinese labs currently dominate. American open-source models have thus far been unable to compete at the open frontier of neither capability nor cost. This gap is a profound failure of the American AI ecosystem and is one of the greatest, least understood risks to U.S. national security. However, recent work from Thinking Machines and other U.S. open-source AI labs offer promising signs, and we hope this marks the beginning of a sustained American response.

Why is the lack of a frontier American open-source model dangerous? A foreign-developed open model could contain a sleeper agent planted by or on behalf of an intelligence service. Copies of the model could spread throughout the global software ecosystem and be integrated into code-generation and cybersecurity systems. The model would remain dormant until it detects a trigger event, such as a geopolitical conflict, and would then introduce attacker-specified vulnerabilities into the software it generates. Later, the adversary could exploit those vulnerabilities through conventional cyber operations or autonomous cyberswarms. These cyberswarms would not need to find a way in because the AI systems we used would have already built one for them.

What’s more, many organizations which need AI for cybersecurity are being pushed toward the Chinese AI ecosystem! The restrictions that American frontier labs are placing on the cyber capabilities of their models prevent many security teams from using them for their legitimate security work. In many cases, Chinese open-source models are the only frontier systems willing to do such work. Policies intended to reduce cyber risk may be shifting sensitive cybersecurity work onto models built by America’s principal geopolitical adversary.

Put simply: we are embedding foreign-developed AI systems we do not fully understand throughout our software ecosystem. If even one contains a sleeper agent, we may be installing a latent insider threat across civilian and government networks; one that could remain dormant until an external event activates it.

The lack of a frontier American open-source model is a major risk to U.S. national security.

Call to action

We must build a frontier American open-source model and invent ways to detect AI sabotage

Build a competitive American open-source model. We need an American open-source model that matches China’s best open models. American developers, researchers, and investors must treat this as a national security mission. NVIDIA’s planned Hugging Face acquisition and deals with Poolside and Essential AI are promising, but no U.S. release has yet closed the gap with China. The federal government can support this effort through compute funding and procurement commitments.

Enable controlled distillation for American model developers. Frontier labs should consider lawful distillation agreements with vetted American open-model developers. Chinese labs are already using illicit distillation to acquire American capabilities. Controlled agreements could strengthen domestic models without exposing a frontier lab’s original weights or core intellectual property.

Preserve access for legitimate defenders. Until a competitive American open-source model is available, frontier labs should continue expanding access to their cyber programs for verified defenders. We must also develop more sophisticated safeguards that prevent misuse without making American models unusable for legitimate security work, including for users outside these programs.

Develop methods for detecting model sabotage. Frontier model weights must be treated as critical infrastructure. AI labs, universities, and government agencies should make model forensics a research priority. We need reliable methods for finding backdoors and sleeper agents before models enter sensitive systems and a dedicated program with resources to continually examine popular models. Research directions could include mechanistic interpretability, automated methods for eliciting hidden behaviors, and forensic tools that identify suspicious patterns in internal model representations.

Develop technology to monitor AI models for sleeper-agent activation. AI security companies, labs, and government agencies must build tools that allow organizations and individuals to continuously monitor deployed models for dormant malicious behavior. Publicly available weights are not proof that a model is safe.