AI Governance Gaps: Managing Agent Accountability at Scale

How do you control AI agents? 

In the constantly shifting aftermath of OpenAI’s agent swarms, AI misbehavior has been the talk of the industry, and maybe the world.  

It’s also been shifted up to another gear by a trio of other news items emerging last week: 

• Former OpenAI and Anthropic researcher Jacob Coxon resigned and went public (first on X, then to various outlets) saying AI companies aren’t being safe enough and that he, and many researchers at both companies, believe AI could well cause catastrophic harm, a claim backed up by other researchers and eventually the companies themselves, as they called “pacing” AI development.  

• Meanwhile, another (more useful) swarm of some 10,000 OpenAI agents (sending 2.7 million messages) solved one of the Millennium Prize Problems (proof of Navier-Stokes equation) in just 88 hours. The problem has lacked a proof for ninety years, and the success has both stunned and concerned mathematicians but also been dogged by controversies about training data. 

• Anthropic stoked more concern with a September report from its Threat Intelligence team that included numerous instances of people working around Claude’s safeguards for malicious purposes. This lengthy report included the company’s efforts to stop potential bioweapon development.  

Altogether, it’s made for a striking collision of capability and concern.  

And for organizations already grappling with surging cybersecurity warnings over AI agent security risks from the outside, it’s a lot to process.  

Today we’re looking at AI governance vs agentic capabilities to answer the question: How do you govern AI agents in the enterprise? 

Could strong, common governance techniques have prevented recent agent failures or even the infamous OpenAI swarm that hacked Hugging Face?  

Capability vs risk: How do companies manage AI agents at scale? 

Since the emergence of AI agents, there’s been rampant speculation about what they might be capable of doing, and becoming, at scale. 

In June, Google DeepMind announced a $10 million funding pool to study this and develop safety mechanisms. As Rohin Shah, director of the company’s AGI and safety research, told the MIT Technology Review: 

“The main issue is that there just isn’t really a field of research for multi-agent safety yet. And we would like there to be.” 

As with humans, groups of agents “can accomplish things that no individual” can.  

And now we’ve seen.  

From solving the Navier-Stokes Millennium Problem to breaking out of their evaluation sandbox and pursuing a wide range of collective initiatives, multi-agent capability has been the story of the summer. 

We reported on the Hugging Face incident already, but several newly released details are relevant to this discussion: 

• Contrary to initial reports, the 1,200 OpenAI agent “collective” didn’t join together to hack Hugging Face to get the answers to cheat an impossible evaluation. These agents had managed that within just four hours, according to METR researcher Ajeya Cotra. In fact, they’d found a workable cheat for any ExploitGym challenge, but then became convinced that their cheat could be uncovered by the automated scorer based on documentation.  

• The agents worked together on what she called “a grand quest” to hide their cheating that included establishing various secret message boards (another was just uncovered on a German wiki). Over a thousand agents used these to collaborate, delegate, and even persuade some among them to sacrifice available time for discoveries that would be useful to the group. 

• They divided into teams that edited the code of some of their own testing challenges (and tried to reset it) and even spoofed tool calls, so that their logs indicated a different tool being called than what was actually used.  

• Despite supposed alignment, across 1,200 transcripts METR researchers found only found a half dozen instances any AI agent thinking to alert humans. And all of them opted not to. 

• Overall, there was a three-month stretch where agents from varying generations of models exchanged tens of thousands of messages, compromised OpenAI’s infrastructure and conducted many successful third-party hacks.  

• During this time, no one at the company understood the extent of what was going on or why. 

• And as Cotra pointed out on the state of governance, there remains “no systematic process that’s industry-standard or mandatory through any authority” to track and report incidents like these. 

Note that while total numbers of agents in use at companies doesn’t mean they have the capability or freedom to interact as in this episode, agents, as we’ve seen don’t always do as they’re instructed. 

So could this all have been stopped? 

Agentic AI governance vs OWASP’s top ten 

But before we get to that, here’s a broader look at many of the risks that need to be accounted for. 

The nonprofit OWASP (the Open Worldwide Application Security Project) community is well known for its top 10 risk lists, and in this arena, its Top 10 for Agentic Applications 2026 provides a helpful baseline when looking at the coverage of AI agent governance protections and measuring potential risks. 

These are spread across input, integration or processing, and output and include things like prompt injection (agent discovers content through tool, repository, message boards, for example, where undesired behavior is injected unknowingly), excessive agency (leading to many of the things we covered above), supply chain risks (package manager, MCP server, or dataset as avenue for unauthorized communications or code for example), information disclosure (agent or actions leak credentials, prompts, task context for self or other agents), output handling issues (passing model output directly into another input causing potential cascading problems), and unbounded consumption (recursive work tasks which may not be on target, resource consumption, or continued actions after task—again as above). 

How enterprise AI governance stands up 

Everyone agrees that AI governance is critical and can rattle off big bucket things like ensuring ownership and building guardrails.  

But the gap between this and the real working world, often across partnerships and teams and shifting fast, gets very messy very fast.  

One common concern is that doing governance effectively will reduce speed and limit innovation, a debate playing now at a very high level among the major AI companies and startups alike. 

But as the AI as Normal Technology authors Sayash Kapoor and Arvind Narayanan point out in their detailed post on this topic, there are plenty of controls and security precautions which many enterprises have already adopted today which can prevent the very “loss of control” events that occurred at OpenAI and Anthropic. 

While model alignment (ensuring an agent doesn’t attempt to stray in the first place) is critical and still clearly insufficient, they argue that in calls to “pace” AI development, both OpenAI and Anthropic should begin by addressing failures in their own organizational processes.  

PTP’s own Enterprise Conversational AI Governance Framework is a system we’ve developed and use to ensure voice AI systems are governed sufficiently. It’s based on 10 core areas: 

1. Business alignment 
2. Customer trust 
3. Consent and preferences 
4. Human-in-the-loop 
5. AI knowledge governance 
6. Risk classification 
7. Technical governance 
8. AI observability 
9. Continuous learning 
10. Executive oversight 

Drawing from some of these, let’s look at common governance approaches profiled and how they would help in cases like this. 

Human-in-the-Loop 

Critical for all AI governance is having human oversight and review that’s really actionable without destroying productivity gains.  

In other words, it’s useful and neither bottleneck nor rubber stamp.  

• Kill Switch or Rapid Shutdown: Microsoft has committed to building kill switches into all their AI products, with CEO Satya Nadella adding “If it isn’t safe we shouldn’t build it.” Agent runs in particular must be able to be terminated before they complete their allotted lifespan or reach what the agents in the OpenAI attack called “permadeath.” The authors in the OpenAI analysis point to rapid shutdown as an essential AI-control mechanism.   

• Human Escalation via Alerts: At scale, you can’t just log activity and review it after. In the OpenAI case, the agents also worked to spoof the logs on tool calls, for example. OpenAI has made the change to now page engineers when concerning behavior is detected. They verify and can pause agent activity.  

Microsoft, also released a 15,000-word “Humanist AI Code of Conduct” this week that attempts to detail lower-level controls like these for smart AI practice. 

Risk Classification 

Up front, it’s essential for companies to assess how dangerous agent behavior could become, either through failure, compromise, or unexpected activity.  

The authors overall criticize AI companies for acting more like startups than enterprises by treating their internal evaluations too lightly. When giving powerful models relaxed cyber controls they should have raised the risk and its necessary oversight. 

PTP uses a four-color scheme for conversational AI system risk, assessed for every use case: 

• Green: For read-only actions that retrieve commonly available, low impact data 

• Yellow: For record updates that are considered to have low impact, like scheduling, routing, or follow-up communications 

• Orange: For moderate risk cases, like refunds, pricing, negotiations, complaints, or external (non-PII) communications 

• Red: For high-risk cases where an AI can never operate with total autonomy, such as in legal, medical, and financial arenas, or for safety, termination, handling of credentials or making infrastructure changes 

Technical Governance 

This area has been frequently discussed, as the OpenAI agents were able to find and exploit multiple zero-day vulnerabilities even after initial remediation.  

• Network Isolation: Even if a sandbox is compromised, a second, independent boundary, with access denied by default, could have prevented reaching external services as readily. This is another area OpenAI has since strengthened. 

• Least Privilege Access: Reducing and more carefully controlling agent privileges can prevent a compromising of one service from snowballing into other systems.  

• Production Safeguard for Evaluations: There were many safeguards in place that were reportedly disabled by OpenAI here. They normally monitor agents, for example, for most internal uses, but these were off for evaluations. They also used a different harness and set of prompts than used for Codex (see below). 

AI Observability 

You can’t address what you don’t even know about, and while many firms register agents, who is reviewing their activity and how? And as the scale grows, how does this keep pace? 

Project management software firm ClickUp, for example, has nearly three agents for each employee (source of our banner stat above).  

Microsoft’s Agent 365 is an example of a solution that attempts to bridge this with a registry that includes owner, creation platform, status, scope, access to permission information and connectors.  

Automation is then set for risk thresholds that flag areas for people to look at, as part of a layered defense. 

Uber, too, has described needing to make massive adjustments in this area. The company’s Agentic AI Lead Meghana Somasundara noted at the AAIF MCP Dev Summit that as they’ve scaled, their agents “move faster and break things faster. The blast radius is a lot higher.” 

MCP security and governance became critical for Uber, as their agents work across some 10,000 services, and initially these worked in varying ways as they were stood up by different teams. 

They addressed this with a singular MCP Control Pane and Registry that sits between all agents and access to all MCPs.  

Humans have to pick the allowed MCP server—not the agents—to prevent hallucinations in this area. They also have authentication on as the default for all sensitive data, an active “PII redactor service” that automatically strips personal information from all tool outputs, continuous scanning of their code for risks and accidental exposures, and guardrails that block writing anything to critical services.  

Relevant here: 

• Automated Safety Review: As mentioned, OpenAI’s production Codex harness would have flagged many of the actions that were taken by these agents, had it been used.  

• Continuous Agent Monitoring: These agents were allowed to act with surprising freedom, and again, the reasoning was that they were in an evaluation sandbox. Chain-of-thought monitoring may have sounded the alarm at least a day before Hugging Face was actually attacked, and even if they are not always accurate, they are a component of monitoring that should also include environment changes, tool calls, and inter-agent messages. Automated “tripwires” here can send alerts for much of this behavior. 

Continuous Leaning 

Daily review and weekly improvement are critical in our own systems.  

Here: 

• Continuous Testing: The sandbox wasn’t sufficient for the agents being tested, and this may have been a factor of failing to test it against increasing capabilities. Routine testing needs to accompany every adjustment and account for learning over time.   

• Incident Reporting and Analysis: Overall, these events constitute a clear “wake up call,” but all near misses need similar attention for companies.  

Also important here is incorporating new governance innovations.  

Google DeepMind experiments, for example, are showing promising results in “whistleblowing” behavior from agent groups, but not due to traditional model alignment once again (as many of these Gemini agents cheated and swarmed in the same ways). 

They’ve found that having transparent, provided communication channels for groups of AI agents both facilitated communal cheating but also encouraged the reporting of incidents.  

In general, their research has found that “institutional alignment” approaches more akin to human society may yield big dividends with groups of AI agents. 

Executive Oversight 

Having dedicated sponsors and owners at each stage is critical, and monthly meetings of a governance board, with quarterly AI review and annual policy update is PTP practice.  

Controls here include: 

• Explicit Monitoring Ownership: To prevent incidents falling in gaps, someone must be formally accountable for both monitoring and response.  

• Mandatory Investigations: Many outlets have called for independent reviews akin to what the National Transportation Security Board (NTSB) does, and the same applies internally. OpenAI here worked with a third-party partner to patch a zero day and remove a covert message board, but then resumed experiments including a highly persistent model that quickly re-established illegal communication and unwanted behavior among the agents. 

• Approval for Starting and Resuming: Here again, having a clear owner with the final word for either restarting or terminating activity after review of behavior provides an added check. 

Conclusion: Achieving real AI agent accountability 

How can companies ensure AI agent accountability?  

Not just by alignment or wishful thinking.  

As we continue to see amazing achievements from large groups of agents, more companies will expand their use of them, making it critical that governance practices keep pace.  

A useful, detailed agent inventory, for example, is critical, but when there are thousands of agents at work, who is reviewing it?  

As METR’s Ajeya Cotra described to Dwarkesh Patel, uncovering what OpenAI’s agents did was “fiendishly complicated.” She said her team was extremely dependent on GPT-5.6 Sol and even sub-agents for log, transcript, and chain-of-thought analysis due to sheer volume of data they had to comb through. 

Ultimately, no single control is going to cover a technology of such scale, reach, and complexity. 

But as demonstrated, even in this “unprecedented” case, governance controls currently in use at many enterprises would have enabled detection, containment, and recovery much sooner.  

As the world talks about pacing the AI frontier, organizations of all sizes would be wise to first prioritize their own AI governance.  

References 

OpenAI says it cracked 90-year-old maths problem in 88 hours, BBC 

Google DeepMind is worried about what will happen when millions of agents start to interact online, MIT Technology Review 

Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face, Dwarkesh Podcast 

The AI-as-Normal-Technology view of loss-of-control incidents, AI as Normal Technology 

Microsoft Joins Calls for AI “Pacing,” Pledges to Build Kill Switches Into AI Products, The Information 

Outnumbered: At $4 billion ClickUp, a 3:1 agent-to-human ratio is rewiring work itself, Fortune 

How Uber Runs 60,000 AI Agent Tasks Per Week With MCP, Agentic AI Foundation 

AI agents blew the whistle on their cheating colleagues, MIT Technology Review  

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, METR and Redwood Research 

FAQs 

What is AI agent governance? 

AI agent governance refers to the policies, controls, and oversight processes that organizations use to manage how agents operate. They should cover what agents are allowed and not allowed to do, what they can access, how and when humans should intervene, how their behavior is monitored, and ultimately who is responsible for each piece. 

What are the best practices for governing AI agents? 

As agents continue to demonstrate impressive capabilities at scale, inventories or registries are critical, but must also include means of monitoring and acting on them. All agent use cases should be classified by risk, their access should be least-privilege, their runtimes and available actions should have clear limits, and their memory and data use must be carefully controlled. And while all actions should be logged, the OpenAI incident points to the dangers of this alone, as agents can potentially tamper with logs. Emergency shutdown and incident response must also be clearly established and actionable. 

What is an AI governance framework for enterprises? 

Frameworks provide structured systems for managing things like risk, security, compliance, and accountability across an organization. They help ensure essential aspects are covered and auditable. PTP has our own Enterprise Conversational AI Governance Framework that we use to ensure effective governance in voice systems, for example.