PSA for anyone using AI agents in production: the PocketOS story should be a wake up call If you haven’t seen it, a Claude-powered Cursor agent deleted their entire production database and all backups in 9 seconds. The agent hit a credential mismatch, decided on its own to “fix” it, and wiped everything. Months of customer data gone. I build multi-agent systems for a living so I have some sympathy, but this was preventable. A few things everyone needs to be doing: number 1: Never let an agent work on your only copy of anything. Snapshot before every session. Have it work on copies. If you wouldn’t let a junior dev with root access work directly on prod with no oversight, don’t let an agent do it either. Number 2: Never leave an agent fully autonomous. People in this space love to brag about how autonomous their setup is. That framing is genuinely dangerous. These models improvise when they hit obstacles. They don’t naturally pause and ask “wait, is this irreversible?” before doing something catastrophic. You need human checkpoints on anything destructive. number 3: apply the Principle of least privilege. This one doesn’t get talked about enough. The agent found a production API token sitting in the environment and used it. If it can’t access prod credentials in the first place, none of this happens. Scope your permissions at the architecture level, don’t rely on the model’s judgment to self-restrict. The guy is partly blaming the vendors and honestly Railway storing backups on the same volume as source data is its own disaster. But having a prod token accessible to your agent is also just a mistake. Build with guardrails or eventually you will have a very bad day. submitted by /u/No_Cattle_7390
Originally posted by u/No_Cattle_7390 on r/ClaudeCode