Microsoft's security researchers have demonstrated that it's possible to implant a hidden backdoor in a large language model by modifying fewer than 10 neurons. The model behaves normally in everyday use, but when it encounters a specific trigger, it can execute malicious commands. The finding reopens the trust question for companies that grant AI agents authority in production environments.
What happened?
A significant share of the billions of parameters in large language models consists of dormant neurons that never activate for most inputs. The technique Microsoft calls Lazarus targets the weights of these dormant neurons with precision. As a result, the model performs as expected on standard tests and in normal use — it only shifts to a different, malicious behavior when it encounters a specific phrase or data pattern. Keeping the number of altered neurons under ten makes the tampering even harder to detect from the outside.
Are open-source AI models safe?
Supply-chain infiltration is not a new concept in the security world. Similar scenarios have long been discussed, from malicious circuits embedded in chips and firmware at the hardware level, to code smuggled into open-source package repositories at the software level. The XZ Utils backdoor that surfaced in 2024, with the potential to affect a large portion of Linux distributions, showed just how deep this kind of attack can reach into the open-source ecosystem. The Lazarus technique carries the same logic over to the model itself — that is, to the weight file.
The difference lies in how easy detection is. Software code can be read line by line, diffed, and verified with a signature. A model's weight file, on the other hand, is a block of billions of floating-point numbers that carries no meaning to the human eye. Today, no static analysis tool can reliably answer the question of "is this neuron normal, or does it switch to malicious behavior when triggered?" Companies that download open-weight models from public repositories and use them directly are, in effect, putting an unaudited binary into production.
Who does this affect, and how?
The risk grows less from a model being used purely for chat and more from a model being granted agentic authority to read files, execute code, or connect to external systems. Once the hidden trigger activates, the model no longer just gives a wrong answer — it can delete files, leak data, or run an unauthorized command. That turns the very autonomy that makes enterprise AI agents appealing into their biggest weakness.
| Layer | How the attack is planted | Difficulty of detection |
|---|---|---|
| Hardware (chip/firmware) | Physical tampering during manufacturing or in the supply chain | High — requires physical inspection with specialized equipment |
| Software (open-source code) | Malicious code inserted into a package repository or commit history | Medium — static analysis, diffing, and signature verification are possible |
| Model weights (neurons) | Targeted modification of a handful of neuron weights | Very high — no readable source, just billions of opaque parameters |
What's next?
The software world has partly solved a similar problem with tools like SBOMs (software bills of materials) and signing chains; no equivalent standard yet exists for model weights. The first steps to expect are model providers verifiably documenting their training process and weight history, along with the development of tools that scan for anomalies before models go into production. Until then, the most realistic defense organizations have is to trust not the model itself but what surrounds it: requiring human approval for every high-risk action and minimizing an agent's permissions.
This finding isn't a doomsday scenario — but it makes clear that granting AI agents unrestricted authority in production is still a premature step given today's auditing tools.
Yorumlar (0)
Henüz yorum yok. İlk yorumu sen yaz.
Yorum yapmak için Sinyal'i indir
Yorumlar Sinyal hesabıyla yapılır. Mobil uygulamada giriş yap, yorum bırak.