just a quantified reports based system with geographics information and subscription tier so that we can get some statistics coming in about what time of day and point in the release cycle the provider quality is at, and letting us see what the current model quality is at before wasting time (and tokens) trying to corral a quantized model. I can’t seem to find this information aside from just random complaints on reddit surfacing occasionally, which for some reason doesn’t seem like a reliable indicator running a few obfuscated benchmarks at random times throughout the week would also give us some metrics about how the provider models are actually doing in a way that providers can’t just dismiss it as subjective (which is not the case; I use the same set of prompts to open a development cycle and some days the model just has no idea what to do with it and ultimately fails to do anything useful) submitted by /u/probablyblocked
Originally posted by u/probablyblocked on r/ClaudeCode