Today’s leading public coding benchmarks are starting to saturate at the frontier: top models cluster within a narrow score band where adjacent configurations often overlap on confidence intervals. DeepSWE is a long-horizon software engineering benchmark built to separate them. It delivers four advances over existing public benchmarks: Contamination free : Tasks are written from scratch, not adapted from existing commits or PRs, so no model has seen the solution during pretraining. High diversity : Tasks span a broad pool of 91 repositories across 5 languages. Real-world complexity : Prompts are ~half the length of SWE-bench Pro’s, yet solutions require 5.5x more code and ~2x more output tokens. Reliable verification : Verifiers are hand-written to test software behavior rather than implementation details. The result is a benchmark that reflects how today’s frontier coding agents actually perform in software engineering work. Source: https://deepswe.datacurve.ai/ submitted by /u/GroundFlock
Originally posted by u/GroundFlock on r/ArtificialInteligence