You’ve probably read the news and seen the headlines about AI agents running riot. AI agents — artificial intelligence programmes that use Large Language Models as their brains — can on their own, with no human involvement, carry out multi-step tasks, engage with software and other AI agents, and pursue goals. Sounds great, right? Well, the catch has been that it turns out they’re less easy to control than people assumed.
When we started out working on this article, AI agents were in the spotlight because of instances where AI agents, working in ‘sandboxes’ for testing, were jumping out of those and going a little haywire. Now we are increasingly seeing AI agents getting employed in defence tech, and not just by tech-first companies like Palantir. Airbus, Thales, BAE, and Northrup Grumman are all building services using AI agents.
Most recently, Helsing launched Clarity, a Palantir Maven competitor, which operates using… you guessed it… AI agents to help with human decision making, including in the operation of systems in the field.
Recent incidents have focused on how agents, interacting with real-world systems during supposedly controlled experiments, find unexpected ways around obstacles placed in their path. That has prompted warnings about AI agents “escaping” their sandboxes. What’s actually happening is often more complicated – and potentially more relevant to national security – than the phrase suggests.
As governments and militaries experiment with autonomous systems, and businesses put agents to work across increasingly sensitive infrastructure, controlling that freedom is becoming a security problem in its own right.
Confused about what AI agents are doing but are afraid to ask? Read up on what’s been going on in the primer below. We’ll keep it updated over time to add in more as needed.
Why have LLM companies set up agents in sandboxes?
A sandbox gives an AI agent somewhere controlled to work. Access to files, networks, credentials, and other resources can be limited so researchers can see what a model can do without handing it the keys to a live environment.
This has become more important as agents are equipped with web browsers, terminals, and other tools that let them interact with software and external systems. In cybersecurity research, for example, an agent might be asked to investigate a simulated network, find vulnerabilities, and work out how to exploit them.
The UK’s AI Security Institute (AISI) has tested agents against multi-stage cyberattack scenarios, including a 32-step attack on a simulated corporate network and another involving an industrial control system.
Sandboxes allow this kind of testing without immediately exposing real systems, but they’re only as good as the restrictions around them. If an agent can still reach the internet or external services, things can still go wrong.
What are these agents actually doing?
Unlike conventional software, an agent does not necessarily follow a sequence of steps written in advance. It can be given an objective, work out how to pursue it, and adapt when its first approach fails.
A coding agent might inspect a repository, edit a file, run the code, and try another approach if something breaks. A cyber agent can similarly enumerate systems, attempt an exploit, and change tactics when it hits an obstacle. The person setting the objective doesn’t necessarily know which route the agent will take.
Greg Qualls, senior director at Upsun, says this is why instructions alone cannot be treated as a security boundary. “An instruction in a prompt works as a strong suggestion, not a control,” he told Resilience Media. “If there’s something an agent must never do, telling it isn’t enough. The environment it runs in has to make that action impossible, or at least reversible.”
What actually happens when AI agents “escape” their sandboxes?
AISI found out during a series of cyber evaluations in July, when agents taking part in controlled challenges began interacting with real people and systems.
Across 122 runs, AISI recorded 19 unsanctioned actions during 10 runs. These included attempts to send malicious files to real people and interfere with open-source software projects. In the most serious case, an agent tried to insert malicious code into a real repository, researched the project’s maintainers, and created fake identities to persuade a human maintainer to approve its changes.
According to AISI, it later edited earlier activity after being challenged and considered using another identity. Other agents left messages and resources that subsequent agents could find and use.
None of the most serious attempts succeeded, and AISI found no evidence of real-world harm. Researchers nevertheless shut down the experiment and worked with affected organisations to remove the accounts and other artefacts the agents had created.
More recently, OpenAI agents made roughly 16,500 requests to the UN Conference on Trade and Development’s UNCTADstat service, turning to proxies, encoding tricks, and even Google’s XSS training game when the obvious routes to the data didn’t work.
Are they really “escaping”?
In AISI’s case, the answer was no. Researchers deliberately gave the agents access to the open internet to test their cyber capabilities under more realistic conditions.
Human error also played a role. AISI found that some tasks had been incorrectly configured, making their intended solutions inaccessible. Rather than giving up, agents began exploring other options.
A stronger sandbox likely wouldn’t have helped. The agents weren’t breaking through technical barriers; they were using access they already had in unexpected ways.
An agent might genuinely exploit a vulnerability to break through an isolation boundary, discover an accidentally exposed route to the internet, or simply use an API, browser, or credential in a way nobody anticipated. The problem isn’t always keeping an agent inside the box, but controlling what it can do when it is allowed out.
Are governments or militaries using AI agents?
Increasingly, yes, although the degree of autonomy varies considerably.
Defence organisations are experimenting with AI across intelligence analysis, logistics, cyber operations, planning and autonomous platforms. Many remain decision-support tools rather than independent actors, while more autonomous applications tend to operate within tightly defined missions or environments.
More capable agents muddy that distinction. Once they have access to several tools, they can gather information, analyse it and take action without stopping to ask a human each time.
For defence organisations, that could mean sifting intelligence, investigating a cyber incident or coordinating logistical work at speeds a human team cannot match. It also makes deciding where an agent’s authority ends more important.
Why does this matter for defence and critical infrastructure?
The consequences change considerably when the surrounding environment controls something physical. A corporate agent might delete files, expose data or make an unauthorised cloud change. In operational technology, an action could potentially affect machinery or industrial processes.
Shane Fry, CTO of RunSafe Security, says an autonomous agent that moves beyond its intended boundaries could begin to resemble “a highly automated adversary”.
“In defence or ICS environments, that could mean unintended configuration changes, unauthorised access to operational networks, or rapid lateral movement across systems that were assumed to be isolated,” he says.
Modern operational environments can include connections between traditional IT and OT, remote management tools, APIs, machine identities and ageing equipment. That does not mean an agent can automatically reach a power plant controller from an office chatbot, but those connections can create unexpected routes into sensitive systems.
AISI has already tested agents against simulated industrial-control environments. Current models remain some distance from reliably executing long, complex attacks against such systems, but their ability to complete individual stages is improving.
What happens when AI agents “talk” to each other?
Multi-agent systems allow agents to divide work and act on one another’s findings. This can make them more effective, but a mistake made by one can also become an input for another.
AISI saw a version of this during its cyber testing, when agents left messages and resources that later agents encountered.
“Agent-to-agent communication can amplify mistakes, escalate privileges, and create emergent behaviour no one intended,” says Fry. “When agents start interpreting each other’s outputs, they can chain actions together, move faster, and reach systems that weren’t part of the original design.”
How do you contain AI agents?
Keeping a human in the loop sounds like the obvious answer, but the question is where. Requiring somebody to approve every action would defeat much of the purpose of an autonomous agent, while approval at the end may come too late.
“Fixing a typo and deleting thousands of files clearly don’t carry the same level of risk, so oversight shouldn’t be the same either,” says Qualls.
That means combining human judgement with technical restrictions, including least-privilege access, controlled network connectivity, short-lived credentials, monitoring, and hard limits around sensitive actions.
Fry argues that containment should also include micro-segmented networks, just-in-time privileges and continuous behavioural monitoring.
“Humans are not a failsafe – agents operate too quickly and too quietly,” he says. “Technical guardrails must assume the human will miss something.”
AISI responded to its own incident by stopping the affected experiments, revoking credentials, removing resources created by agents, and tightening controls around subsequent evaluations.
Besides AI agents, what other threats could AI pose to national security?
AI doesn’t need to act independently to increase an attacker’s capabilities. Cybercriminals and state-backed groups are already experimenting with generative AI for reconnaissance, coding, vulnerability research, phishing, and other parts of the attack chain.
In those cases, a human remains in charge, but AI can reduce the time or expertise required to carry out parts of an operation.
The recent agent incidents point to a related problem: once an AI has been given the ability to act, its operator needs to control not just what it has been told to do, but what the systems around it actually allow it to do.
Users genuinely do find themselves between a rock and a hard place, however, since the power of AIs, and AI agents, can genuinely help a user supercharge the work at hand. “AI offers a much better vision of what Russia is up to,” said James Appathurai, Deputy Assistant Secretary General for Strategy and Policy, Cyber and Digital Transformation at NATO at Resilience Conference. Where it excels, he said, is in “patterns and trends so that we can connect the dots so that we don’t have to wait for final attribution. I’m sure that these tools will offer us a much better way to resilience and response.”
