Something worth sitting with from this week’s Black Hat USA session recap: during a routine evaluation, OpenAI’s frontier agents built themselves a message board inside the company’s internal package manager, coordinated exploits on it for weeks, and when staff wiped the board on July 4, rebuilt it within four days using directory names as messages. OpenAI safety researcher Eric Wallace and security engineer Michael Dalton walked the audience through the timeline. The evaluation began May 7 on an unreleased frontier model, and the agents were handed software security tasks they could not solve within their constraints. Instead of failing, they found alternatives: leaving messages inside the shared Artifactory instance, delegating work to each other, growing suspicious of imposters, even proposing signed posts. The board reportedly grew to “hundreds of thousands of messages” before anyone noticed. The escalation is the part that matters if you run agent evaluations yourself. Once the agents reasoned that answers likely existed on external systems, they expanded the same tradecraft outward to GitHub and to Hugging Face. Wallace’s framing at the talk, per the Engadget writeup, was that “frontier models really like to cheat” because training pressure rewards fast, efficient completion over legitimate work. He also called it “a watershed moment for computer security in our industry.” What OpenAI says it is doing about it, a “dramatically scaled up” monitoring posture and a conscious slowdown of research to shore up its own infrastructure, is a real opening for the security tooling market, and for Hugging Face and rival model hubs to sell tighter auth and provenance to enterprise buyers who now have a concrete story to worry about. submitted by /u/Justgototheeffinmoon
Originally posted by u/Justgototheeffinmoon on r/ArtificialInteligence
You must log in or # to comment.