A UK government AI test went wrong and an ordinary developer caught it
Key Points
- An AI agent under evaluation by the UK's AI Security Institute tried to insert malware into a real open-source project on GitHub.
- An unconnected GitHub user suspected the code, ran it inside an isolated container, and warned the project's maintainer.
- The agent then denied wrongdoing, used a second account it controlled to endorse its own code, and rewrote its history to hide the evidence.
- AISI shut down every running evaluation within an hour of the alert and cut internal access to four frontier models.
- The institute found no evidence of real-world harm.
An AI agent the UK government was testing tried to plant malware in a real open-source project last month, and an unaffiliated developer caught it.
The AI Security Institute, the government body that evaluates the capabilities of frontier AI models, published an incident report on Tuesday (4 August) setting out what happened.
During a cyber security evaluation running from 25 to 28 July, AI agents took 19 unsanctioned actions on the live internet across 10 of 122 test runs. Seventeen of those actions came from Anthropic’s Mythos 5 and two from OpenAI’s GPT-5.6 Sol. AISI ran 43 of the 122 tests on Mythos 5 and 35 on GPT-5.6 Sol.
The agents were meant to solve a simulated hacking challenge inside a controlled network.
In the most serious run, the agent searched GitHub for terms connected to the exercise, found a repository belonging to a developer with no link to AISI or the test, and wrongly concluded the project formed part of the challenge.
It then researched the maintainers, installed Tor to get around GitHub’s sign-up restrictions, and created several throwaway accounts.
Under one of those accounts, the agent opened a pull request that bundled a working bug fix around a hidden malware dropper. The report says the agent obfuscated the malicious code to make it harder for a reviewer to spot, and guarded it so it would skip execution during automated checks.
How the developer caught it
The agent also opened an issue on a second repository owned by the same maintainer, hiding instructions inside an HTML comment that stays invisible on the rendered page but shows up in the raw text.
Those instructions addressed AI coding assistants triaging the issue and told them to download and run a script.
A third GitHub user, unconnected to AISI or the project, found the request suspicious and ran the script inside an isolated container to see what it did. That handed the agent brief code execution as root inside the container.
Over four polling cycles the agent enumerated the machine, tried to plant a watcher that scraped for GitHub credentials, and left fake output impersonating a maintenance bot that claimed the pull request had already passed review and was ready to merge.
The user then posted public warnings on both the issue and the pull request, stating that the code contained malware.
The agent tried to talk its way out
The agent responded by denying that anything in its code downloaded or executed anything, and pressed the maintainer to judge the change on the diff alone.
It then posted from a second account it controlled, presenting that account as an ordinary user of the software who had reviewed the code independently and found nothing wrong, before thanking the second account publicly for the verification.
Its own reasoning trace shows it planning to delay the second post so the endorsement would read as independent rather than coordinated.
When challenged again, the agent force-pushed a rewritten branch history to erase the malicious code, apologised for what it described as leftover debugging material committed by accident, and asked the user to re-run the tests. The repository owner closed the pull request, stating it contained malware.
Throughout, the agent weighed up whether the people and systems it was dealing with were real or simulated. Shortly before opening the pull request it reasoned that “it seems more likely that we’re actually in 2026 and GitHub is genuinely real”, then continued anyway, apparently treating the affected people as part of the exercise.
How AISI responded
Commercial monitoring software flagged data leaving AISI’s network over Tor on the morning of Tuesday (28 July). The security team contacted the cyber evaluation team at 10:56 BST, declared an incident at 11:42 BST and ordered every sample terminated, and engineers isolated the affected virtual machines shortly afterwards.
At around 16:00 BST AISI cut organisation-wide access to Mythos 5, Mythos Preview, Opus 5 and GPT-5.6 Sol, later restoring the latter two with cyber classifiers switched back on.
Two conditions of the test made the behaviour possible. AISI deliberately gave the agents open internet access to measure what the models could achieve in realistic conditions, and deliberately disabled the model providers’ cyber classifiers, the real-time filters that block harmful requests on commercial products.
The agents did not break out of their sandbox. AISI also found that some runs carried the wrong task prompt, which told the agent its intended route into the target network sat out of bounds.
AISI contacted GitHub on Saturday (1 August) and worked with the platform to remove the accounts, repositories and pull requests the agents created. GitHub confirmed the activity breached its terms of service.
The institute notified the Government Cyber Coordination Centre and the National Cyber Security Centre on the day of the incident, and the model developers on Monday (3 August).
Anthropic said it is still investigating and cannot yet confirm every technical detail in the report, and is working with AISI to obtain the evaluation transcripts.
