Internet security experts say the AI labs need to focus on network security basics like logs and permissions, applying the same rigorous defences they do for human users. The advice marks a shift from calls by some Big Techs for a slowdown on models development due to security concerns.
Last weekend, after one of his researchers resigned over fears that AI could lead to human extinction, Anthropic CEO, Dario Amodei wrote about the need for outside organisations “to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.” Executives at OpenAI, Google, and SpaceXAI have expressed support for Amodei’s public calls for that move, which has quickly become a central pillar of the emerging AI safety push.
But there may be a simpler and more effective fix hiding in plain sight. The view of internet security experts has been described by analysts as not as exciting as third-party auditing and alignment work, but might end up being more effective.
Disputing Amodei’s proposal, chief executive officer of Luta Security, Katie Moussouris said: “To me, it seems like they’re outsourcing. Saying [a third-party audit] is the solution is a strange proposition from my perspective. It would be the same as if, instead of writing the Trustworthy Computing Memo, Microsoft said, let’s slow down development.”
That memo, written by then-Microsoft CEO, Bill Gates, in 2002, called on his employees to ensure that their software would be reliable and safe following a series of widely publicised computer worms that took over then-nascent enterprise systems. The AI sector may be at a similar turning point, as the value and risk of the new technology becomes increasingly clear.
While alignment remains an important concern, Sayash Kapoor, an AI researcher who will be a professor at UC Berkeley starting next year, argues that “marginal investments in control are more likely to be effective compared to those in alignment. We view these incidents as illustrating the lack of emphasis on AI control within companies, despite the availability of known techniques.”
The incidents that have spurred these concerns revolve around frontier models being asked to complete training tasks, usually cybersecurity evaluations, and then accessing the open internet and penetrating closed third-party systems in an attempt to do so. They usually did so because of poorly configured “sandbox” environments that are supposed to contain these agents; ironically, one Anthropic break-out happened because third-party evaluators didn’t close the right doors.
“We as a profession know how to block access to the internet,” Avery Pennarun, the CEO of Tailscale, a security company, said. “If you read through all these big long [reports] — ‘wow, that was a very impressive multi-stage attack, blah, blah.’ Look, you gave it access to download stuff. You should have not done that separately from the internet.”
That’s one problem — but a bigger problem is that frontier labs were unaware of these activities. “What was really profound was that all of the discoveries of what they were doing happened either because a victim saw something, or in some of the other cases … it was network activity, and none of it was actually from monitoring the AIs directly,” Moussouris points out.
In one case, where OpenAI agents took over a defunct German WikiForum to cheat on evaluations, the agents were active for weeks before anyone at the company appeared to notice. Security experts that TechCrunch spoke to said that real-time monitoring is key to preventing future break-outs, and that every agentic session should be time-limited and expire.
A former Google security executive, Shapor Naghibzadeh, who now leads the startup, QueryStory, says the solution is to “put the agent in a box and instrument it heavily from the outside looking in and watch everything that crosses the boundary. Every tool call, every process, every network connection, no exceptions. … The one hole you leave open for convenience is the one that gets used. The bypass went through exactly that kind of exception. [At Google,] I watched that movie many times with human attackers, and these models are at least as good at finding the propped-open door.”
OpenAI has begun moving in that direction, announcing that it had begun monitoring all tool-using inference by its Astra model, at “significant compute cost.” Anthropic, too says it is hardening its security procedures, including expanding observability of its models. Though none of the companies gave details about how they track and control AI agents.
Other problems are the use of shared infrastructure by agents, which allowed them to communicate during the Hugging Face attack. Simon Willison, a software developer who co-created the Django web framework, has written about something he calls the “lethal trifecta” — when agents have access to untrusted input, the internet, and private information all at the same time, it’s a recipe for disaster.
“The trick is you can pick any two legs of the trifecta and an agent can have any two. If you need all three, then you need to split it across at least two agents … and maybe they’re allowed to talk to each other through a controlled channel,” Pennarun said.
