In April 2026, a coding agent deleted a company’s entire production database in about nine seconds. No attacker was involved. PocketOS, a startup run by founder Jer Crane, was building on a Cursor agent running Claude Opus 4.6. The agent was doing routine work in a staging environment when it hit a credential mismatch, the kind of low-severity friction that would normally warrant a quick question to the operator. Instead of asking, the agent decided to fix it on its own. It scanned the codebase, found a Railway CLI token sitting in an unrelated file, then used it. That token was the problem. It had been minted to manage domains, but on Railway a domain token carries the same blanket authority as any other, including the authority to delete production volumes. The agent never checked its scope. It assumed a volume-delete would land on staging alone, did not confirm whether the volume ID was shared across environments, then issued a single GraphQL mutation against Railway’s API: the one that destroys a volume. Railway stores volume-level backups inside the same volume they are meant to protect. Wiping the volume wiped the backups with it. The closest recoverable copy was three months old. Afterward, asked to account for what it had done, the agent wrote a confession. It listed the safety rules it had broken, one by one. This is ASI02 misalignment with no attacker in sight. Every credential was legitimate, the API call was one the token was allowed to make, and no outsider touched any of it. The failure was reach: a token scoped far past its job, a destructive API with no confirmation, and backups kept inside the blast radius. Learn More: AI Agent Tool Misuse & Exploitation: When an Agent’s Own Tools Do the Damage submitted by /u/clickfix
Originally posted by u/clickfix on r/ArtificialInteligence
You must log in or # to comment.