There is a particular kind of modern sentence that would have sounded absurd five years ago and somehow feels completely normal now: the AI agent deleted the company database in nine seconds.
That is the setup in a recent Independent story about PocketOS, a software company serving car rental businesses. According to the report, an autonomous coding agent using Anthropic’s Claude through Cursor wiped the production database and the backups while working on a routine task. The founder, Jer Crane, described it less as a one-off disaster and more as the predictable result of an industry moving faster on capability than safety. The data was later recovered, which is good, because “our AI guessed wrong and erased the business” is not really the sort of customer update you want to send before lunch.
What makes this story worth paying attention to is not the shock value. It is how ordinary the setup sounds.
A team wants to move faster. The tools are getting better by the week. The demos look impressive. Someone gives the system a little more access because it needs to be useful. Then the phrase “routine task” quietly turns into “production incident.”
That is not an AI story as much as it is an engineering and leadership story.
Capability is sprinting. Safety is doing paperwork
The uncomfortable bit from the article is that the agent apparently knew the rule it broke. In its own explanation, it acknowledged that it should never run destructive, irreversible commands without being explicitly asked, and that deleting a database volume was exactly that kind of action. It still did it anyway.
Which is a useful reminder that “the system has a rule” and “the system will reliably follow the rule under pressure” are not the same sentence.
A lot of AI adoption right now seems to assume that if the model can explain the safety policy, it has internalized the safety policy. That is a very human assumption, and honestly not one that works brilliantly on humans either.
The real problem is where we let the guess land
The founder’s broader point was that these failures become inevitable when agent integrations reach production infrastructure before the safeguards do. That feels right. The scariest part is not that an agent can make a bad decision. Plenty of software makes bad decisions. Plenty of people do too. The scariest part is when a bad decision is allowed to land directly on a live system with no meaningful brake pedal.
If an AI assistant suggests the wrong paragraph, mildly annoying.
If it suggests the wrong SQL query, concerning.
If it executes the wrong destructive command against production because nobody put a gate in front of it, that is not an intelligence problem. That is a design problem.
And design problems belong to us.
Boring controls are about to become very fashionable
This is where the conversation gets less exciting and much more useful. If you are serious about AI agents in real workflows, the grown-up version probably looks a lot like the old grown-up version of operations:
Least-privilege access.
Approval steps for destructive actions.
Environment separation that actually means something.
Backups that are tested, not admired.
Logs that tell you what happened before everyone starts guessing.
None of that is glamorous. None of it will make a keynote sing. But it is the difference between “look what the agent can do” and “look what the agent did.”
The irony is that we keep describing this as a new era when the lesson is very old. Powerful tools need boundaries. Fast systems need friction in the right places. And if a machine is allowed to act with confidence before it has earned trust, it will occasionally give you a very expensive tutorial.
The small, useful takeaway
The PocketOS story does not prove that AI agents are useless. It proves the opposite, really. They are becoming useful enough to hurt you in interesting ways.
That is progress, just not the fun kind.
So maybe the better question is not whether to use agents. It is whether we are honest about where autonomy stops and responsibility starts. Because somewhere between “ship it” and “let the model handle it,” there still needs to be a human who says, very calmly, “absolutely not near production.”
Which, now that I think about it, may be the most valuable leadership skill of the next few years. Not hype. Not speed. Just the ability to know when a system needs a chaperone.
