Your Team Feels Faster. The Stopwatch Disagrees.
Vishal Sachar
Co-Founder & CEO of CLRT
Ask a team that has adopted AI whether it is faster and the answer arrives instantly and with conviction. Ask a stopwatch and the answer is less comfortable. In the only randomised trial to time experienced developers on their own real codebases, the feeling and the measurement did not merely diverge. They pointed in opposite directions, and the distance between them was roughly forty points.
The study, published by the research group METR in July 2025, is hard to dismiss on the usual grounds. It was not a synthetic benchmark and it was not a survey. Sixteen experienced open-source maintainers worked through 246 real issues in repositories they knew deeply, with each task randomly assigned to permit or forbid AI tools, and each one timed. Before starting, the developers forecast that AI would speed them up by 24%. When AI was allowed, tasks took 19% longer, with a confidence interval running from +2% to +39%: the entire range is a slowdown. The finding that matters most, though, is what happened afterwards. Having just lived through the slower tasks, the same developers estimated that AI had sped them up by 20%. The perception error was roughly forty points wide, and direct experience did not correct it.
The story since then is not a clean redemption arc, and the messiness is the point. METR's follow-up, published in February 2026, expanded to 57 developers, 143 repositories and more than 800 tasks. This time returning developers completed tasks 18% faster with AI and new developers 4% faster. But both confidence intervals cross zero, from -38% to +9% and from -15% to +9% respectively, which means neither result can be distinguished from no effect at all. Worse, 30 to 50% of participants quietly withheld tasks they refused to attempt without AI, which bends the sample, and METR itself describes the new data as only very weak evidence. The honest reading of the best research available is that the slowdown has probably eased and may have reversed, and that nobody, including the people running the most careful studies in the field, can currently measure the effect well.
Set that fog against what people say about themselves. In METR's May 2026 survey of 349 technical workers, the median self-reported speedup from AI was 3x. Not a third faster. Three times. This is the same population whose self-assessment, in the one setting where it was checked against a clock, ran about forty points hot. And yet self-report is precisely the input most companies are using for real allocation decisions: which team gets agent licences, which workflow gets automated next, what the AI line in next year's budget should be. A founder steering by how fast the team feels is steering by the one instrument the controlled evidence says is broken.
None of this is an argument that the gains are not real. It is an argument about where real numbers come from. The largest randomised trial of a workplace AI tool, run across 56 firms and drawing on Microsoft's own telemetry, found that Copilot users spent about 30 minutes less on email each week, completed documents 12% faster, and that only around 40% of workers given licences used the tool regularly at all. Notice the texture of those findings: specific, bounded, and smaller than the story. That is what measurement sounds like. The companies that will actually capture AI's value are not the ones that feel fastest. They are the ones instrumented well enough to know: a measured baseline before adoption, evals that define what good looks like before an agent runs, and gates that check the output instead of trusting the sensation.
A team that feels three times faster and measures 19% slower is not lying to you. It is uninstrumented.
A deeper dive
It is worth understanding why the illusion is so durable, because it is not carelessness and it will not be fixed by telling people to estimate harder. AI redistributes where the effort in a task sits. It compresses the parts that feel like work, the blank page, the boilerplate, the first draft, and it expands the parts that barely register as work at all: reading generated output, spotting the subtle wrongness in it, re-prompting, integrating, repairing. The developers in the METR trial spent less time writing and more time reviewing and correcting, and reviewing simply does not feel like time in the way writing does. Effort and duration are different quantities, and human memory reports effort. So the honest introspection of a capable professional returns that was much easier, and the mind rounds it to that was much faster, even while the clock disagrees. This is also why the error survives experience. Every day with the tool refreshes the feeling of ease, and the feeling of ease is the very signal being misread.
The second thing worth internalising is that measuring this properly is genuinely hard, which is exactly why casual internal evidence is worse than useless. METR's follow-up is the most careful attempt anyone has made, and it was still bent by its own participants: a third to half of them withheld tasks they were unwilling to attempt without AI, a selection effect no statistics can fully repair, and one of the reasons METR grades its own data as very weak evidence. If the best-resourced measurement effort in the field struggles, the pulse-check survey your team ran in a spreadsheet is not evidence. It is the perception error with a chart attached. What works instead is instrumentation built into the work itself: a baseline captured before the tool arrives, evals that define correct output for the specific workflow and run on every change (the argument of Evals Are the New Tests), and verification gates that pass or fail work product on criteria that exist outside anyone's feeling of speed. Instrumentation like that does not just tell you whether AI helped. It tells you where it helps, which is the question the budget actually needs answered.
Key terms
- Randomised controlled trial
- A study that assigns tasks or people to conditions at random, so a measured difference can be attributed to the tool rather than to who chose to use it.
- Confidence interval
- The range of effects a study cannot rule out. An interval that crosses zero means the data cannot distinguish the result from no effect at all.
Work with CLRT
CLRT builds the instrument. Before we automate anything we baseline the workflow it replaces, define the evals that decide what counts as good, and put verification gates between agent output and anything that matters, so the question of whether this is actually faster has a numerical answer instead of a mood. If your AI budget is currently allocated by feel, bring us one workflow and we will put a stopwatch on it.
Vishal Sachar
Vishal Sachar is the Co-Founder and CEO of CLRT, where he helps UAE businesses make sense of applied agentic AI and put it to work. He writes on agentic systems, AI governance, and the economics of automation. Reach him at vishal@clrtstudio.com or on LinkedIn.
