I was at a conference – I forget which one, as I’ve been to over 12 of them this year – talking to a Digital Trailblazer about his AI agent deployments. He said he had over 1,000 deployed to production.
I was floored. I was even more shocked reading that Gartner predicts an average Fortune 500 enterprise will have over 150,000 AI agents in use by 2028.
150,000. Let that sink in. It reminds me of the projections in the 1990s about how many websites and domains we’d see on the internet, the 2000s when we estimated how many mobile applications we’d see in app stores, or the 2010s estimates of IoT devices.
I think the 150,000 number is exaggerated, unless Gartner includes sub-agents and micro-agents in that number. But even at 100 agents, certainly by 1,000, executives must have an operating plan for managing and governing AI agents.
Michael Allen, CTO at Laserfiche, says, “As AI agents move from experimentation into critical workflows, organizations are responsible for establishing a clear operating model that governs how agents access information, make decisions, and perform tasks.”
Are AI Agents delivering value? Or like kindergartners causing mischief?
Now when I think of AI agents at scale, several security, data, and IT disciplines come to mind. Every agent needs an identity and defined access privileges. Companies will need to establish a brain for the AI agents – a context layer- and then establish strong AI/data governance around its information, IP, and other critical data assets.
CIOs should be very careful about deploying more than a handful of AI agents without these architectures and practices.
But when I picture thousands of AI agents, I’m concerned. I’m frightened that many of them act like kindergartners running around the schoolyard. Picture that, and you’ll understand why enterprises must decide who’s accountable when they have thousands of AI agents in production.
5 Operating principles to institute before scaling AI Agents
Here are five operating principles that CIOs must address when scaling somewhere between 10 and 1,000 AI agents.
1. Assign a human owner to every agent
A child has a parent and a dog has an owner. Inside companies, business leaders should have clearly defined decision authorities. My first two assertions are generally true. But in many companies, who decides what and where accountabilities lie is rarely defined with clear-cut answers.
Saying a C-level has accountability – the CFO owns finance, etc. – that’s too high-level. A RACI around initiative accountabilities and responsibilities – only the program managers with their elaborate spreadsheets ever understood them.
We need clear-cut definitions of who’s accountable for an AI agent. Don’t deploy one without this defined.
“Every agent needs a human owner, a defined scope of intent, a behavioral baseline, and continuous monitoring that flags deviation before it becomes an incident,” says Marshall Heilman, CEO at DTEX. “Orchestration multiplies the decision points where drift can happen, and each needs governance for the agent’s life.
Let’s be honest, but we screwed up with data owners. We let business users deploy dashboards without someone owning the underlying data. Don’t allow this to happen again with AI agents.
(Note: StarCIO workshops combine AI Strategy and Governance to ensure business leaders can’t pursue their AI vision without owning the AI agents they want to deploy.)
2. Define when humans step in
At last week’s Coffee With Digital Trailblazers on AI agent strategy, we discussed three types of AI agents:
- Human augmentation agents, which are built to partner with people to complete their objectives. My best example is IP creation. Whether you’re developing an app or writing a book, chances are the AI agents working should be collaborating with you, not generating, finalizing, and publishing the IP without your review.
- Human-in-the-middle agents are designed to make decisions and take actions, but have decision boundaries for when to have people review them.
- Agentic, which I equate to largely autonomous agents. MIT quotes research from Horton defining agentic as, “autonomous software systems that perceive, reason, and act in digital environments to achieve goals on behalf of human principals, with capabilities for tool use, economic transactions, and strategic interaction.”
During the episode, I shared five levels of AI agent autonomy and simplified and aligned them with industry terminology.
The debate over the intent, authority, and terminology will continue, but for now, let’s simplify. Organizations must define people versus AI agent responsibilities. We’ll need instances of when people must step in to review an AI’s decisions or actions before they’re taken. We’ll also need to consider times when speed-to-decision is critical, and AIs need to act without waiting on people.
“Governance can’t sit outside the agentic operating model as a separate review process; it has to be built into how agents are orchestrated, monitored, and allowed to act,” says Vivek Vaidya, co-founder and CTO at Kana. “That means defining clear ownership, decision rights, and escalation paths so humans know when to intervene and who is accountable when an agent behaves unexpectedly.”
3. Measure AI and human agents
Quantifying people’s contributions and productivity has always been challenging. But we can’t deploy AI agents at scale without a defined approach to validating success criteria, measuring business value, and quantifying risk.
“Measurement is a prerequisite for AI agent orchestration, and the standard must be the same for AI and human agents,” says Rob Scudiere, CTO at Verint. “Systems must include mechanisms to score AI and humans with the same success criteria and quality management programs because a lower bar for AI hides problems instead of fixing them. Once you can see that clearly, orchestration becomes the back office acting on the information, routing work up front and refactoring what isn’t working.”
“You can’t improve what you don’t measure.”
Somewhere between 10 and 1,000 AI agents, CIOs will need to consider implementing AI orchestration platform(s) to scale and govern MCP and A2A. According to KPMG’s AI Quarterly Pulse Survey Q3/2026, 25% report actively developing multi-agent systems, compared to 6% the previous quarter.
Another note: There are over 65 “AI Orchestration” platforms, or platforms that advertise these capabilities. Our research on AI Orchestration platforms is coming out soon.
4. Plan for agents going off course
You can send an ill-behaving kindergartner to the principal’s office. But finding rogue AI agents is a challenge, and even harder is recognizing model drift or when agents start veering off course.
“In most companies over the next year, agents going off course might look like an eager intern with too much access, says Sriram Iyer, SVP and general manager, AI strategy and products at OneTrust. “They’re taking an otherwise approved action in the wrong context, making decisions with outdated data, using inherited permissions too broadly, or passing a mistake to another agent.”
Iyer recommends business functions assign clear ownership for every agent and workflow, establish rules for permissions and access, determine where human approval is required, and maintain continuous controls that allow, block, or escalate actions while maintaining an audit trail.
Gou Rao, co-founder and CEO at NeuBird AI, says, “Knowing an agent is misbehaving starts with looking beyond the output and watching its reasoning, because a confident wrong decision looks identical to a right one until something breaks downstream. True agentic governance demands end-to-end explainability, which requires tracking the agent’s decision path against the business workflow with verifiable citations tied to every intermediate step.”
AI agent owners must anticipate their agents going off course, and before deployment, they should describe how to recognize the signs. These can be implemented as evals and monitoring alerts. They also need a backup plan for when this happens. Then, add recognizing and recovering from AI agents going rogue to your business continuity plan. EY reports that 26% of organizations using agentic AI admit they can’t detect unauthorized AI agents operating internally.
5. Trace agent decisions from prompt to action
While general AI laws exist, most regulators, including those in the U.S. and EU, haven’t specified how they will audit AI agents or what agent-specific controls enterprises must have in place for semi-autonomous agents. CIOs should get ahead of the regulators by defining agent observability standards and establishing AgenticOps practices.
“When an agent behaves badly, businesses need to be able to reconstruct the full trace, from its prompt and reasoning to the context it retrieved and the actions it took,” says Brian Verkley, director, AI data platform at VAST Data. “That gets very difficult when data, agent runtime, telemetry, and policy are spread across disconnected systems, forcing teams to correlate them before they can understand and respond to what happened. AgenticOps will ultimately be about using that operational record to understand agent behavior, enforce boundaries, and decide when a human needs to step in.”
Barr Moses, co-founder and CEO at Monte Carlo, adds, “Most enterprises can tell you what their agents are supposed to do, but almost none can tell you what an agent actually did an hour ago. Agenticops has to close that gap.”
Perhaps it’s time to focus on quality over quantity. A lot of work is needed before scaling to thousands of AI agents.
Leave a Reply