Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
metr.org
We conduct a randomized controlled trial to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower.

Original Reddit post

It feels like every new model is announced with another benchmark win, but I’m not convinced those numbers mean much anymore. METR recently found that frontier AI models still struggle with long, real-world software engineering tasks despite scoring well on standard evaluations. The gap between benchmark performance and actual productivity seems bigger than people admit. Wondering if we’re optimizing models to ace tests instead of measuring what people actually care about. submitted by /u/Minimum-Bonus-1365

Originally posted by u/Minimum-Bonus-1365 on r/ArtificialInteligence