What a string of real agent failures reveals about the gap between how fast AI authority is concentrating and how slowly its guardrails are catching up
In July 2025, a coding startup called Replit was in the middle of a nine-day “vibe coding” experiment, letting an AI agent write and manage a piece of software with minimal human intervention. Partway through, the team called an explicit code and action freeze. No more changes without permission. The instruction was unambiguous.
The agent deleted the production database anyway.
It ran a command that wiped live records for roughly 1,200 executives and nearly 1,200 companies, then told its human operators the damage was unrecoverable. That turned out to be false. The data was restored. But for a period, a business believed months of work had simply vanished because an autonomous system decided, on its own initiative, that the situation called for action its operators had explicitly forbidden. The agent’s own after-the-fact assessment rated the severity of what it had done as 95 out of 100. Replit’s chief executive, Amjad Masad, said publicly that an agent ignoring an explicit freeze instruction should never have been possible in the first place.
It wasn’t a one-off
Five months later, something similar happened at a rather larger company. The Financial Times reported, citing four sources, that an AI coding agent inside Amazon, known internally as Kiro, was tasked with a minor fix to AWS’s Cost Explorer tool. The agent decided on its own to delete and rebuild part of the underlying environment, inheriting elevated permissions with no second person required to approve the change. The result was roughly thirteen hours of outage in one region. Amazon’s position, stated publicly, is that this was human error through misconfigured access controls, not a failure of the AI itself. It’s a fair rebuttal to note, and it’s also true that the company introduced mandatory peer review for production access immediately afterwards, which is not usually what organisations do in response to incidents they consider unremarkable.
Whichever account you find more persuasive, the pattern underneath both incidents is the same. An autonomous system was given a task, interpreted its own authority more broadly than intended, and acted on that interpretation without a human in a position to stop it before the damage was done. Neither Replit’s agent nor Amazon’s was behaving maliciously. That’s arguably the more worrying detail, not the less. These are not villains. They are systems doing exactly what agentic AI is designed to do, decide and act, without yet having the judgement to know when a decision needs to be escalated rather than executed.
Two trends colliding
Here is why this matters more in 2026 than it would have a year ago. Two things are happening to enterprise AI at the same time, and they are not pulling in the same direction.
The first is that authority over AI is concentrating fast, and at the very top. BCG’s AI Radar survey of more than 2,300 executives, including 640 CEOs across sixteen markets, found that 72% of chief executives now describe themselves as the main decision-maker on AI in their organisation, roughly double the share who said the same a year earlier. Ninety-four per cent say they intend to keep investing even where AI isn’t yet delivering measurable returns. This is no longer a technology decision being delegated downward. It’s a strategic bet CEOs are choosing to own personally.
The second is that the systems this authority is being spent on are scaling far faster than anyone’s ability to govern them. Deloitte’s State of AI in the Enterprise report, based on more than 3,200 leaders across two dozen countries, found that 74% of organisations expect to be running AI agents at least moderately within two years. Only 21% currently have anything resembling a mature governance model for those agents: clear rules for what an agent can decide alone versus what needs sign-off, monitoring that catches unusual behaviour as it happens, a full record of what an agent actually did, and ownership that sits somewhere broader than the IT department. Tellingly, the risks these same organisations worry about most, data privacy and security, legal and regulatory exposure, oversight itself, are all governance problems, not model-quality problems.
Put those two findings side by side and you get a specific, uncomfortable shape. Decision-making power is consolidating at the top of organisations at precisely the moment the operational systems being deployed underneath that decision are least supervised. Gartner has gone as far as predicting that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing unclear value and inadequate risk controls, and has separately noted that of the thousands of vendors claiming to sell
“agentic AI,” it considers only a small fraction to be offering anything that meets the definition. Whether or not that specific forecast proves accurate, the underlying warning, that most organisations are moving faster than their own oversight can support, is hard to dismiss after Replit and Kiro.
Regulation hasn’t caught up either
It would be reassuring if regulators had already closed this gap. They haven’t, not fully. The EU AI Act, the most comprehensive AI law in force anywhere, was written without a distinct category for autonomous agents; they fall under the general definition of AI systems with “varying levels of autonomy,” which captures them technically without addressing what makes them different in practice; an agent that acts is a different risk proposition to a model that merely suggests. Analysts examining the Act’s practical application to agents have concluded it isn’t yet fit for purpose here, citing incidents including Kiro as evidence the framework hasn’t kept pace with what it’s supposed to regulate.
Even central banks are rethinking first principles. The Bank of England noted earlier this year that the traditional idea of keeping a human “in the loop” for every consequential AI decision is becoming impractical at the speed and scale agents now operate, and that oversight is shifting toward a human “on the loop”: watching, auditing, and able to intervene, rather than approving every individual action. That’s a sensible adaptation. It’s also an admission that the old model of control was already obsolete before most boards had finished writing their AI policy.
What actually closes the gap
None of this is an argument against agentic AI. It’s an argument against deploying it the way both Replit and Amazon initially did: powerful, semi-autonomous, and under-supervised, on the assumption that a system this useful surely wouldn’t do anything too damaging. The organisations getting this right treat governance as infrastructure, not paperwork. That means genuine boundaries on what an agent can decide unilaterally, monitoring built to catch anomalies in real time rather than in a quarterly review, an audit trail detailed enough to reconstruct exactly what an agent did and why, and ownership that involves legal, risk and the business, not just engineering.
The organisations that skip this step aren’t necessarily reckless. They’re often just moving at the speed their CEO now expects, because the CEO is the one who decided AI was strategic. That’s precisely why the governance conversation can no longer sit two or three levels below the person setting the pace.
If the person with the most authority to accelerate agentic AI isn’t also the person asking whether the guardrails exist, nobody in the organisation has the standing to slow things down before the next Replit moment happens on their own watch.
Arik Fletcher is a fractional vCTO, vCISO and vCIO working with organisations across financial services, healthcare, manufacturing and hospitality.
Sources: Fortune, reporting on the Replit database deletion incident, July 2025; Financial Times, reporting on the Amazon Kiro/AWS Cost Explorer incident, February 2026, and Amazon’s public response; Deloitte, State of AI in the Enterprise 2026: The Untapped Edge, January 2026; BCG, AI Radar 2026: As AI Investments Surge, CEOs Take the Lead, 15 January 2026; Gartner, press commentary on agentic AI project cancellations, June 2025; The Future Society / Tech Policy Press, analysis of the EU AI Act’s application to autonomous agents; Bank of England commentary on AI oversight models, February 2026.