Remember That Study That Scared AI Fans a Year Ago? It Just Spectacularly Fell Apart.
In the summer of 2025, a rigorous randomized controlled trial (RCT) from METR — a respected nonprofit focused on evaluating frontier AI systems — dropped like a bombshell. It involved 16 experienced open-source developers working on their own large, real-world repositories (often over a million lines of code). Tasks were randomly assigned: some with AI tools (mostly Cursor Pro + Claude 3.5/3.7 Sonnet), others without.
The slowdown contradicted not just developer optimism but also expert forecasts from economists and ML researchers.
It was the kind of careful, contrarian evidence that fueled debates: maybe AI coding tools weren’t the productivity rocket fuel everyone hoped for, at least not yet, in high-stakes, high-quality open-source work.
Fast Forward to Early 2026: The Sequel Nobody Saw Coming
METR tried to run a follow-up with newer models, more participants (dozens more developers), and a broader set of repositories. They aimed to track how the productivity picture was evolving.
- Recruitment and retention became much harder. More developers simply wouldn’t agree to do 50% of their work in the “no AI” control condition.
- 30–50% of participants admitted they were selectively withholding tasks — avoiding submitting ones where they expected big AI wins, just to dodge the manual version.
- One developer put it perfectly: “My head’s going to explode if I try to do too much the old fashioned way because it’s like trying to get across the city walking when all of a sudden I was more used to taking an Uber.”
The experiment didn’t collapse because of bad methodology (though there were other issues like lower pay rates contributing to selection bias). It collapsed because the control group became nearly impossible to populate. Participants literally voted with their feet — or rather, with their refusal to submit tasks.
METR was admirably transparent about this. Their raw data from the follow-up hinted at speedups (e.g., around 18% for returning participants, though confidence intervals crossed zero), but selection effects made the results unreliable. They’re now redesigning their approach entirely.
What This Really Means
But the broader takeaway is even more powerful: AI adoption has crossed a threshold where “doing it the old way” feels intolerable to many experienced users.
The world changed faster than the preprint cycle. What felt like a cautious, even pessimistic result in mid-2025 became outdated by early 2026 — not because of one flashy new model, but because habitual use reshaped what developers consider baseline effort.
This pattern isn’t unique to coding. As tools improve and integrate deeply into workflows, the psychological and practical cost of going without them rises sharply.
Control groups in productivity studies become ethical and logistical nightmares.
Also read:
- Bending Spoons: The Italian “Software Surgeon” That Revives Old Internet Brands — and Just Had a Massive IPO
- BlackBerry’s Renaissance: Why the Stock Is Surging Again (and QNX Is the Real Star)
- How’s Allbirds Doing After Its Wild AI Pivot? Now It’s Smartbird — and the Sneakers Are Officially History
- How to Find a Startup Idea — Practical Advice from Y Combinator
The Lesson for AI Skeptics and Enthusiasts Alike
When arguing about AI’s impact, check the expiration date on your citations. A study from two years ago (or even one year ago) may reflect a different technological and behavioral reality. Capabilities advance, user expertise grows, and agentic workflows evolve. Yesterday’s slowdown can become today’s “I can’t imagine working without it.”
METR deserves credit for rigorous science and honest communication. They didn’t hype a reversal — they admitted the experiment broke because reality moved on. That kind of integrity is rare and valuable.
In the end, the most telling data point wasn’t a percentage. It was developers saying, in so many words: We’re not going back. The Uber is here, and walking across town just isn’t an option anymore.
Subscribe to our newsletter
Get the latest Web3, AI, and crypto news delivered straight to your inbox.