The METR study: AI made developers 19% slower while they felt 20% faster

In mid-2025 the research organisation METR published a result that got a lot of attention: experienced developers using AI coding tools took 19% longer, even though beforehand they expected to be 24% faster, and afterwards still believed they had been 20% faster. The number has been quoted both by people who say AI doesn’t help and by people who say the study is wrong. This article covers what the trial did, what it can and cannot tell us, what the 2026 follow-up found, and how to borrow its method to measure your own work.

Facts checked against: METR’s blog post and paper on arXiv (2507.09089), and METR’s update of 24 February 2026, as of October 2026.

What the trial did

METR ran a randomised controlled trial (RCT), the most reliable design when you want to know what causes what.

• People: 16 developers, each a long-time contributor to a large open-source project. The repositories averaged over 22,000 GitHub stars and over a million lines of code.
• Tasks: 246 real issues from their own projects (bug fixes, features, refactors), about 2 hours each on average.
• Randomisation: each task was randomly assigned “AI allowed” or “AI not allowed”.
• Tools: mostly Cursor Pro with Claude 3.5/3.7 Sonnet, frontier models at the time.
• Measurement: screen recordings plus self-reported time. Developers were paid $150 an hour.

What gives the result weight is randomising per task: same person, same project, the only difference being whether AI was allowed.

The result

Number	Value
Developers’ forecast beforehand	24% faster
Economics experts’ forecast	39% less time
ML experts’ forecast	38% less time
Actual	19% more time (confidence interval +2% to +39%)
Developers’ belief afterwards	20% faster

The most interesting part is not the 19% but the gap between perception and the clock. The people doing the work, knowing they were being measured, still got the direction wrong.

Worked example: the gap on a 2-hour task

Take an average 2-hour (120-minute) task:

1. Actual time with AI: 120 × 1.19 = 142.8 ≈ 143 minutes
2. Perceived time (believing 20% less): 120 × 0.80 = 96 minutes
3. Gap: 143 − 96 = 47 minutes per task

In their minds they saved 24 minutes; in reality they lost 23. At three such tasks a day, the unnoticed gap is 47 × 3 = 141 minutes, over two hours.

Feeling faster is not evidence of being faster. If you want to know, time it.

Where the time went

The paper analysed the screen recordings. With AI, developers spent less time actively coding and searching, and more time prompting, waiting on the AI, reviewing and fixing its output, and idle. Some figures from the paper:

• Developers accepted under 44% of AI generations; the rest were discarded or rewritten
• About 9% of time went to reviewing and cleaning up AI output
• About 4% of time was spent waiting for generations

The researchers examined 20 factors that might explain the result and found evidence that 5 likely contribute:

1. Over-optimism about AI, using it even where doing it by hand was faster
2. Developers already knew their repositories extremely well, leaving AI little to add
3. Large and complex repositories
4. Low AI reliability
5. Implicit repository context (rules, conventions) that the AI did not know

Number 5 matches our own experience. Part of the fix is writing project rules down for the AI, see handoff files between AI sessions; the other part is reviewing systematically, see a checklist for reviewing AI-written code.

What the study does not show

METR lists explicitly that the result does not show that:

• AI fails to speed up many or most software developers
• These developers or repositories represent most software work
• AI does not speed up work in domains other than software development
• AI in the near future will not speed up developers in this exact setting
• There are no better ways of using AI in this setting

Other limits: 16 people is a small sample, the volunteers may not be typical, and learning effects beyond roughly 50 hours of Cursor use could not be ruled out.

The February 2026 follow-up

METR ran a larger round starting in August 2025: 57 developers, 143 repositories, over 800 tasks. It now labels the original results page as out of date.

Group	Result (negative = less time)	Confidence interval
Original study (early 2025)	+19%	+2% to +39%
Returning developers (10)	−18%	−38% to +9%
New developers (47)	−4%	−15% to +9%

That looks like a flip to a speedup, but METR itself calls this round an “unreliable signal”. The main reason: between 30% and 50% of developers said they chose not to submit some tasks because they did not want to do them without AI, so the tasks where AI helps most dropped out of the data. METR’s conclusion is that developers are probably faster in early 2026 than in early 2025, but the data is only very weak evidence of by how much, and it is redesigning the experiment.

Measure your own work without fooling yourself

You don’t have to wait for the next study. Apply the same idea to your own tasks:

1. Before each task, write down how many minutes you think it would take without AI
2. Randomise whether this task uses AI (flip a coin, or python -c "import random; print(random.choice(['AI','no AI']))")
3. Time it for real, from start until it passes review, not just until the code is written
4. Take the ratio of actual ÷ estimate for each group and compare

A worked example (made-up numbers to show the arithmetic, not a measurement): 20 tasks, 10 per group.

• With AI: estimated 600 minutes total, took 660 → 660 ÷ 600 = 1.10
• Without AI: estimated 600 minutes, took 570 → 570 ÷ 600 = 0.95
• Compare: 1.10 ÷ 0.95 = 1.16 → the AI group was about 16% slower

Dividing by the up-front estimate is how METR adjusted for task difficulty, so a few hard tasks landing in one group don’t skew the result. But 20 tasks is very few: METR used 246 and the interval was still +2% to +39%. Twenty tasks give you a rough direction at best; keep collecting for weeks.

If you are building AI features into a product, the same principle applies: measure with a test set, not a feeling. See a small eval set.

Common mistakes

• Quoting the 19% as proof that AI doesn’t help, when METR says the result is out of date and specific to one group
• Quoting the follow-up as “AI makes you 18% faster”, when the interval crosses zero and METR calls it unreliable
• Timing only the coding, not the review, the bugs that follow, and the waiting
• Choosing which tasks get AI and then comparing. The tasks you pick for AI are the ones you already expect it to handle, so the result always flatters it. Randomise
• Trusting how it felt afterwards, the very thing this study shows can be most wrong

Sources

• Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity — METR
• Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity — arXiv 2507.09089
• We are Changing our Developer Productivity Experiment Design — METR
• AI coding can make developers slower even if they feel faster — The Decoder