I’ve been building a domain-specific MQL5 dataset, and I just published v1.1 of the evaluation. The model isn’t really the product here. I’m using the same 14B base as a measurement instrument to see what changes as the training corpus gets larger and broader. On a private 300-item Expert Advisor holdout: Base Qwen2.5-Coder-14B-Instruct: 1/300 — 0.33% 83k fine-tune: 281/300 — 93.67% 220k fine-tune: 282/300 — 94.00% GPT-5.6 Sol: 286/300 — 95.33% So on EA generation, moving from 83k to 220k barely changed the result. The 220k model finished 1.33 percentage points behind GPT-5.6 Sol on the all-300 headline. I’m not claiming equivalence; the paired difference is not significant at that denominator, and the card reports the alternative denominator as well. The more interesting result was outside EAs. On a separate 200-item holdout covering include files, custom indicators, scripts and services: Base: 56/200 — 28.0% 83k: 135/200 — 67.5% 220k: 168/200 — 84.0% That’s a +16.5 point gain from 83k to 220k, with 49 paired gains against 16 losses (McNemar p = 5.1e-5). That was the result I was looking for: not whether adding more rows could squeeze another point out of an already saturated EA benchmark, but whether the larger corpus actually broadened domain coverage. A few caveats because benchmarks are easy to oversell: Compile success is only a necessary condition. It does not prove the generated code is correct or safe to run. The holdouts are also generator-produced and in-distribution, not human-written production requests. The 184-item public benchmark is still available for anyone who wants to run another model against the published scoring contract. The new 300/200 holdouts remain private, but the per-item hash-keyed results, statistics, model identities, serving configuration and release verifier are published. Benchmark: https://huggingface.co/datasets/CompilingThings/compile-benchmark I’m particularly interested in feedback on the evaluation design and what people would want measured next. submitted by /u/Compilingthings
Originally posted by u/Compilingthings on r/ClaudeCode