I’ve been digging into agent evaluation for setups where the agent writes and runs SQL against a live database (Snowflake, BigQuery, etc.) and shows results to users. The failure mode that seems underserved: the query executes fine and returns real rows just the wrong ones. Wrong join, wrong filter, stale understanding of the schema. Nothing errors, the output looks plausible, but it’s wrong. Static eval sets with prewritten “golden” answers don’t hold up here, because the correct answer changes as the data changes. Interestingly, LangSmith has a cookbook recipe for exactly this storing labels as queries the evaluator runs at eval time to fetch current ground truth but it’s DIY: you build and maintain that evaluator yourself. As far as I can tell, none of the major platforms (LangSmith, Braintrust, Arize) ship live data verification out of the box; online scoring generally falls back to reference-free LLM as judge. I’m considering building a dedicated tool for this: connect your DB and your agent, and the evaluator independently queries the database to verify each output against what’s actually there right now. Before I build anything, I want to know if this is a real problem for other people: If your agent queries a live DB, how do you catch “ran fine, wrong data” failures today? How often does that actually bite you in practice? What’s your current eval stack LangSmith, Braintrust, Arize, custom scripts, nothing? Would you pay for this as a product, or just have Claude Code write you a one-off eval script? If you’d pay, what would make it worth it? If not, why not? Not selling anything. Trying to figure out whether this is widespread before building… submitted by /u/JuniorLeg6988
Originally posted by u/JuniorLeg6988 on r/ArtificialInteligence