AI benchmarks answer a snapshot question: How well did this model perform when it was tested? But for API-served models, there is another question that matters just as much: Does the same model continue to behave the same way over time? We have been continuously evaluating LLMs across coding, multi-turn reasoning, and tool use, and one of the more interesting patterns appeared when we compared variation within a day with variation across days. In one historical analysis covering 31,352 repeated score observations across 49 models , we observed: within-day standard deviation: 2.80 points between-day standard deviation of daily medians: 8.43 points descriptive ratio: approximately 3.01× That does not mean providers were changing their models every day. There are several possible confounders: sampling variation, task composition, missing observations, provider-side serving behavior, infrastructure effects, and changes to the evaluation instrument itself. Daily medians reduce the effect of extreme observations, but they do not eliminate serial dependence or shared day-level effects. The interesting part is that the variation is large enough that treating a model score as a permanent property of the model starts to look questionable. This led us to think about benchmarking less like a leaderboard and more like an observability system. For longitudinal evaluation, we now try to separate several things that can otherwise look identical: Model behavior changed. The model performs differently on the same type of workload. The benchmark changed. A prompt, test, scoring rule, or task definition was modified. The provider changed. Serving conditions, throttling, infrastructure, routing, or backend behavior changed. Randomness happened. Repeated calls naturally produce different outcomes. These distinctions matter because a lower score alone is not evidence of model degradation. One design choice we found particularly important is benchmark versioning . If you change a task, test case, prompt, request policy, or scoring definition, the historical baseline should not silently continue as if the measurement instrument were unchanged. In other words, the useful question is not: It is: We also separate availability problems from capability results. A provider timeout, authentication problem, or transport failure should not automatically become evidence that the model became less capable. Another difficult issue is benchmark contamination . Full transparency sounds ideal, but a continuously operating benchmark has a problem that a static academic benchmark does not: if every live prompt, hidden test, and task variation becomes public, those evaluations can eventually become known to model developers, training pipelines, retrieval systems, or the models themselves. So we have been trying to separate two ideas: Methodological transparency
- people should be able to inspect how measurements are produced, compared, and interpreted. Evaluation secrecy
- some portion of the active task set may need to remain undisclosed or rotate over time if the goal is to continue measuring behavior on effectively unseen workloads. I think this becomes increasingly important as benchmarks themselves become part of the ecosystem models are optimized against. A few questions I’d be interested to hear opinions on: Should major LLM benchmarks increasingly report performance over time , rather than only the latest leaderboard score? How much movement would you need to see before calling something genuine model drift rather than normal variance? Should live benchmark task banks be completely public, partially hidden, or continuously rotated? If the public model identifier does not change, what evidence would convince you that the underlying behavior actually changed? We have published a public technical methodology describing the measurement framework, assumptions, limitations, longitudinal analysis, and reproducibility requirements: https://aistupidlevel.info/asl-public-benchmark-methodology-2026.pdf The public document intentionally does not expose the full live task bank or every operational parameter. Disclosure: I am the founder of AI Stupid Level, the system that produced these measurements. ASL is a commercial platform, but there is no signup, pricing, fundraising, or sales pitch in this post. I’m sharing the measurements and methodology because I think continuous LLM evaluation is an under-discussed technical problem and I’d like the assumptions challenged. submitted by /u/ionutvi
Originally posted by u/ionutvi on r/ArtificialInteligence