AI Coding Assistants Can Raise Bug Rates—But 41% Is Not a Universal Result
AI coding assistants can increase defects without delivering a corresponding speed gain, but the widely repeated 41% figure is not a universal verdict on AI-generated code. It comes from a specific enterprise study announced in September 2024: Uplevel’s Copilot research examined 800 developers, found no significant improvement in pull-request cycle time or throughput, and reported a 41% increase in bugs found within pull requests after Copilot access.
The broader evidence remains mixed. As of August 2026, controlled research still shows that AI can impose substantial prompting, review and correction costs, while newer surveys and preliminary experiments suggest that more capable tools are helping some developers. The practical conclusion is narrower than either “AI writes bad code” or “AI makes everyone faster”: generated code needs to be measured as part of a complete delivery system, including review and rework.
What the 41% finding actually measured
Uplevel compared developers with Copilot access against a control group without access, matching the groups on factors including role, working days and coding volume. Its metrics covered pull-request cycle time, the number of pull requests merged, bugs appearing in pull requests and work outside normal hours. The reported increase therefore concerns bugs recorded in that observed workflow; it does not mean that 41% of all AI-generated code was wrong or that every AI assistant raises defects by the same amount.
The distinction matters because a bug-rate change can reflect several interacting effects. Faster drafting may increase the amount of code submitted for review, reviewers may encounter larger changes, and developers may accept suggestions that fit locally but conflict with assumptions elsewhere in a mature codebase. The study establishes an association within its test and measurement design, not a timeless error rate for every model, language or team.
Its productivity result is equally important. Copilot access did not significantly change either pull-request throughput or the time required to merge a change. That makes the historical finding less about spectacularly broken code than about an unfavorable trade: the measured teams saw more recorded bugs without a detectable improvement in those two delivery metrics.
Newer research makes the picture more complicated
A later randomized trial by METR initially found that experienced open-source developers took 19% longer on tasks when early-2025 AI tools were allowed. By February 2026, however, METR’s follow-up experiment update said its newer data could not provide a reliable estimate of the current productivity effect. The expanded experiment included 57 developers, 143 repositories and more than 800 tasks, but developers increasingly declined tasks that might require them to work without AI, while concurrent agents made time tracking harder. METR considered it likely that tools were producing greater speedups than in early 2025, yet called the new evidence too weak to determine the size of that improvement.
Survey evidence points in a more optimistic direction, although it measures perception and organizational outcomes rather than the same controlled task-completion metric. In September 2025, Google’s account of the 2025 DORA report said nearly 5,000 technology professionals were surveyed: 90% reported adopting AI, more than 80% said it improved their productivity, and 59% reported a positive effect on code quality. DORA also characterized AI as an amplifier of existing organizational strengths and weaknesses, rather than a tool that creates high performance by itself.
These findings do not directly cancel one another. Uplevel observed enterprise engineering metrics around Copilot adoption; METR randomized real tasks completed by experienced contributors in repositories they knew; DORA gathered responses across a much larger professional population. They used different tools, periods, teams and definitions of success. A self-reported productivity improvement can coexist with longer completion time in a controlled task or with more defects appearing during review.
The hidden cost sits between generation and deployment
An assistant can reduce the time needed to produce a plausible patch while increasing work elsewhere. Developers still have to determine whether the change matches the requirement, respects repository conventions, handles failure paths and preserves behavior outside the edited files. If the suggestion is nearly correct, diagnosing its final error may take longer than writing a smaller change directly.
That shifts the useful unit of measurement from “code produced” to accepted, verified change. Suggestion counts, generated lines and acceptance rates describe tool activity, but they do not show whether a team delivered reliable software sooner. A credible evaluation should include downstream work that is easy to omit from a coding-assistant dashboard:
- time from starting a task to merging and deploying it;
- human review time and the number of revision rounds;
- automated-test failures and bugs detected before release;
- escaped defects, reversions and emergency fixes after release;
- the size of submitted changes and time spent understanding generated code.
Teams should also separate results by task type. Generating repetitive tests or documentation is materially different from changing authorization logic, debugging an intermittent failure or modifying an unfamiliar subsystem. Combining those categories into one average can conceal both productive uses and expensive failure modes.
How to use assistants without outsourcing judgment
The safest operating assumption is that AI output is a proposed contribution, not validated code. Keep changes small enough for a reviewer to understand, require the same automated checks used for human-written patches, and ask the author to explain behavior that is not evident from the diff. A passing test suite is evidence about the cases it covers, not proof that a generated implementation is correct.
For a team-level evaluation, define the comparison before beginning the trial. Record a baseline period, identify which developers and tasks receive access, and avoid changing review rules halfway through the measurement. If adoption is optional, note that enthusiastic users may choose tasks where the assistant is strongest; if adoption is mandatory, account for the learning period and for developers who already had substantial experience with similar tools.
Review capacity is another constraint. If assistants let contributors submit larger or more numerous patches, the apparent gain at the keyboard can become a queue for maintainers. Limiting patch size and tracking review latency makes that transfer of work visible. The goal is not to reject generated code automatically, but to prevent rapid generation from overwhelming the slower process of establishing correctness.
What the evidence supports now
The 41% increase remains a legitimate result for the population and metrics Uplevel studied, and its lack of measured speed improvement is central to interpreting it. It should not be promoted into a fixed error rate for all coding assistants. Models, interfaces and agentic workflows have changed, while teams vary widely in codebase maturity, testing quality and review discipline.
Newer evidence does not justify the opposite blanket claim either. METR’s 2026 update explicitly says that widespread adoption has made clean comparisons harder, and DORA’s positive results include self-reported assessments. For engineering leaders and individual developers, the defensible position is therefore conditional: AI assistance may accelerate useful work, but only measurements that include review, defects and deployment can show whether it improved the finished result.
Also read:
Subscribe to our newsletter
Get the latest Web3, AI, and crypto news delivered straight to your inbox.