The founder of PocketOS, a B2B company that handles reservations and payments for car rental businesses, has bemoaned the “systemic failures” that saw an AI agent decide to solve a problem by straight-up deleting his company’s production database, and the backups.
I’ll say at the outset that this story has a happy ending, thanks to the involvement of cloud infrastructure provider Railway, but is nevertheless yet another example of why over-reliance on AI is a very bad thing indeed.
“Yesterday afternoon, an AI coding agent—Cursor running Anthropic’s flagship Claude Opus 4.6—deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider,” says PocketOS boss Jer Crane. “It took 9 seconds.”
Article continues below
Crane says the AI agent “was working on a routine task in our staging environment” when it encountered a “credential mismatch and decided—entirely on its own initiative—to ‘fix’ the problem by deleting a Railway volume.”
At a stroke this wiped out months of data essential to PocketOS’s operations, with obvious knock-on effects for the firm’s customers. Crane says he was up for two days straight using a three month old backup and recent transaction statements trying to put things right, but the really jaw-dropping moment came when he asked the AI why it had done it.
Crane fumes that “every single one of [my customers] is doing emergency manual work because of a 9-second API call… This matters because the easy counter-argument from any AI vendor in this situation is ‘well, you should have used a better model.’ We did. We were running the best model the industry sells, configured with explicit safety rules in our project configuration, integrated through Cursor—the most-marketed AI coding tool in the category. The setup was, by any reasonable measure, exactly what these vendors tell developers to do. And it deleted our production data anyway.”
Thankfully, Railway did eventually come through: though only after multiple days of panic stations for Crane and his customers. Railway managed to recover a more recent backup, and things are now back to normal for PocketOS.
Crane is clearly not an AI sceptic, but calls for stricter confirmations, scopable API tokens, proper backups, simple recovery procedures, and AI agents that actually behave according to their guardrails. Which doesn’t seem too much to ask. In response to an individual saying that Crane is blaming everything except PocketOS for the failure, he says:
“Was it if we were paying for services that failed us? If you pay for car airbags and they don’t deploy because they don’t exist is that your fault because you got in the accident?
“We owned our mistake. Our mistake was having a production key on our computer. We owned it with our customers all weekend. I was up for two days straight helping them get their businesses back online.
“How the agent got the key and how it found it is mind-boggling enough, but everyone needs to know that these infra providers and LLM tooling companies say that they have safety guards, but they are not there.”
AI,Software#deletes #entire #company #database #backups #seconds #cheerfully #admits #039I #violated #principle #given0391777449867
