We evaluate AI agents in production for a living (building Prefactor), so this is something we look at across a lot of different teams’ systems, not just theory. The pattern that shows up again and again: an agent clears every eval in staging, then a few weeks into real traffic it starts drifting. Not crashing, not throwing errors, just quietly doing a slightly different version of its job. It might start answering questions outside its intended scope, handling edge cases inconsistently, or in worse cases touching data it shouldn’t. Standard evals don’t catch this because they’re a snapshot at one point in time against a fixed test set, and production traffic doesn’t look like a fixed test set for very long. A few reasons we keep seeing this happen: Distribution shift in real user inputs versus the eval set, so behavior that was never tested starts occurring more often. Upstream model or prompt changes (yours or the underlying provider’s) shifting behavior in ways a one-time eval never re-checks. Tool and API responses changing shape over time, which agents handle silently instead of failing loudly. Nobody is watching the full run, just aggregate metrics, so the first sign of a problem is a downstream complaint, not a caught deviation. What’s helped the teams we work with is treating evaluation as continuous rather than a pre-launch gate: tracing every run (not sampling), scoring drift and risk in real time, and having a human-in-the-loop option to hold or block a run when something looks off, instead of only logging it for a postmortem later. Disclosure: we build tooling in this exact space (Prefactor), and we’re live on Product Hunt today, currently sitting at #1, if anyone wants to see how we approach it. Mostly curious how others here are handling this, especially if you’re not using a dedicated eval layer. submitted by /u/Diligent_Response_30
Originally posted by u/Diligent_Response_30 on r/ArtificialInteligence