AI Coding Productivity — The Speed Went Up. What Went Down?

Forecast 24, Perceived 20, Measured Plus 19

Sixteen experienced open-source developers tackled 246 real-world tasks. Open-source software is code made public so anyone can inspect and fix it, and the tasks all came from mature projects that these developers had spent an average of about 5 years maintaining. A coin toss decided whether AI tools were allowed or prohibited for each task. The tools were the latest ones available between February and June 2025.

A controlled experiment is an experiment that gives the tools to only one side while keeping the same work, and then compares the results. This has the same backbone as a clinical trial giving a real drug to one group and a fake drug to the other. Using a coin toss to decide which side gets the tools reduces the likelihood that the two groups differed from the start, making it the most rigorous design, and this experiment did just that.

There are three numbers. Before starting, the developers predicted that using AI would reduce completion time by 24%. After finishing, the same developers estimated that AI had reduced completion time by 20%. The actually measured completion time was increased by 19% (arXiv:2507.09089, preprint). This means the tasks that used AI took longer. Because the sample is small, this figure is also an estimate, and the abstract does not state its range. A preprint is a manuscript released before undergoing peer review by an academic journal.

It is worth pausing at the middle number. It is common for predictions to miss the mark. Yet even after finishing all 246 tasks, they still believed they had gotten faster. They did so even while looking back at the work they had done with their own hands after it was finished.

The developers were not the only ones whose expectations missed the mark. Expert predictions gathered in advance for the same study were a 39% reduction from economics experts and a 38% reduction from machine learning experts.

The authors questioned these results themselves. After examining one by one 20 conditions that could have played a role in the slowdown—such as project size, quality standards, and the developers' experience with AI tools—they conclude as follows. While the influence of distortions created by the experimental method cannot be completely ruled out, the slowdown effect appears consistently across multiple analyses, making it unlikely that this was primarily due to the experimental design.

They also note the conditions together. The sample is 16 developers who had some experience using AI, the tools are from the first half of 2025, and the participants are experienced developers well-acquainted with mature projects they have maintained for a long time. This result can only be spoken of within these conditions. The "Speed Went Up" in the title refers to the developers' perceptions and to the side of other experiments and commit log observations we will see later. The actual measurements of this experiment turned out the opposite, and this article starts from the point where those premises diverge.

96 Google Engineers, Opposite Results

Ninety-six full-time software engineers at Google solved a complex, enterprise-grade task—specifically, a task of a scale that would actually be handled inside a company. This was also a controlled experiment split by lottery, comparing conditions using three AI features against a condition using none. It took place in the summer of 2024, about a year ahead of the previous experiment, and the results came out in the opposite direction.

AI significantly reduced the time taken for the task, meaning by an amount that is hard to view as a coincidence. The best estimate, accounting for factors known to influence work time, is a roughly 21% reduction. The authors attached a caveat right within that sentence that the confidence interval is large (arXiv:2410.12944, preprint). A confidence interval is the range where the true value is likely to lie, based on the data. A large range means the actual reduction width could be quite a bit smaller or larger than that. There is also an additional result that developers who spend more time on code-related work during the day were faster when paired with AI.

This is an in-house study where Google measured the performance of Google's internal tool features on Google engineers. The authors also noted its limitations at the end of the paper. They state that the effect size obtained from this laboratory study cannot be assumed to apply directly to a broader range, nor can the effects obtained with Google's internal tool in the summer of 2024 be seen as carrying over to other tools or other times.

The two results split in direction. Placing them side by side shows that what differed was not only the performance of the tools. The people tested were 16 open-source developers with their own projects that they had looked at for an average of 5 years, and 96 full-time Google engineers. The tasks were work on mature personal projects with high quality standards and a complex enterprise-grade task. The tools were general-purpose state-of-the-art tools from the first half of 2025 and three Google internal features from the summer of 2024. The parties doing the measuring differed as well: one side is an experiment outside the tool company (the affiliations of the researchers are not written in the abstract), and the other is a study where Google measured Google's internal tool with Google engineers. Because the people, tasks, tools, and measurers are all different, neither of the two experiments erases the other.

One industry paper summarizes this range in a single line: it reports that controlled studies show a 21% to 56% speed improvement in some environments, and a slowdown of up to 19% in other environments (DOI: 10.58806/ijmir.2026.v3i6n03). This passage does not measure values directly but summarizes previous studies, and that paper marked the improvement claims it made about its own methods as "self-reported." What we borrow from here is only the fact that the range is that wide.

Two things are clarified before reading. First, this article does not argue about which tool is better, nor does it deal with job prospects or copyright disputes. Second, the grades of evidence are uneven. There are many preprints that have not undergone peer review, and some results are models run inside a computer rather than actual observations. Each such instance is indicated within the sentences.

Other Evidence on the Side of Speed

Sixty developers tackled eight enterprise-like tasks under two conditions each. One was without tools, and the other was using tools. They ranged from junior developers with short experience to mid-level and senior developers with long experience, leaving a total of 960 records. Under the tool-using condition, average completion time decreased and implementation errors also decreased. Code clutter and structural complexity also dropped, and even standard survey scores measuring cognitive load—the burden on the mind while working—went down. Maintainability scores, which measure how easily code can be modified later, went up. The benefits were greatest for juniors (DOI: 10.2139/ssrn.6932038, preprint). The experiment was conducted in a single enterprise development environment called .NET.

The final paragraph of the same paper also notes observations from the opposite side. Some of the implementations made by AI inserted unnecessary middle layers (abstractions), made the code verbose, and required additional verification and rewriting. The concluding sentence states that the effect depends on the developer's proficiency, the nature of the task, and whether a human continuously monitors it.

Another controlled experiment dividing 120 professional developers into two groups to analyze 480 code modules also reports both sides together. Average productivity rose by 31.4% and maintainability improved by 18.2%. However, the same code contained 23.7% more security vulnerabilities, which are flaws that attackers can exploit. The variation across languages was also large. Quality improvement was highest in Python (26.3%), while the increase in security risks was highest in C++ (34.8%) (DOI: 10.36948/ijfmr.2025.v07i06.61350, small multidisciplinary publication). The direction aligns with other studies, but there is no basis to trust the decimal places as they are.

A commit is the act of saving by bundling the code modified so far into a single chunk and recording it. It is the most commonly used metric when counting development activity from the outside. The usage of tools has also changed. Agentic coding is a method where, instead of taking dictation line by line, you assign tasks in whole, and the tool independently modifies multiple files and even runs them. The assistant that used to help with dictation has moved toward taking on errands in their entirety.

When analyzing the records of over 100,000 developers active on GitHub—a web service where code is uploaded and edited collaboratively—paired with their AI tool usage records, this generational gap becomes visible. The cumulative increase in commit activity was 40% for autocompletion that fills in code as you type, 140% for conversational coding agents, and 180% for autonomous coding agents (DOI: 10.2139/ssrn.6859839, preprint). The more the tool generation tended to delegate work as a whole, the greater the increase in commits.

Looking only up to this point, the debate seems to be over.

Releases Did Not Increase as Much as Commits

The same study follows that 180% down the production chain one step at a time. The cumulative increase, which was 180% in commits, becomes 50% when counted by the number of projects, and 30% when counted by the actual number of releases put out. All three increased, and the size of the increase gets smaller as it goes down the chain. The authors view this pattern as fitting the weak-link hypothesis. The weak link is the idea that how fast a chain moves is determined not by the fastest link, but by the slowest link. Here, the slow link is the production stage handled by humans.

Code review is the task of having another person read and check code that someone else fixed before merging it into the project. For example, even if the link that writes code becomes two or three times faster, if the links that review, merge, and release stay the same, the output that comes out does not increase by that much. However, this study does not pinpoint which exact stage the human bottleneck is.

The same study estimates the elasticity of substitution between AI effort and human effort to be 0.25. Elasticity of substitution is a value representing how easily one can be swapped out for the other, and the smaller it is, the harder it is to swap them. The authors read this value as meaning that the two are in a relationship where they need each other to generate value rather than substituting for one another. The output side was also checked. On four major app marketplaces, the number of new apps increased to some extent, but total usage did not increase (DOI: 10.2139/ssrn.6859839, preprint). The authors' conclusion is that large gains at the task level have so far only partially translated into software that is released and used. We also disclose the rating. This study is an observational study that later paired and compared people who started using the tool with people of similar past histories, which is weaker evidence than a controlled experiment divided by a lottery.

Another study in the same family adds another layer. Taking the launch time of an agentic tool as a reference point, it followed 61,056 developers whose public commit records bore the mark left by that tool, and 61,056 developers with similar prior activity, from January 2025 to March 2026. The group that used the tool is called the treatment group, and the group taken as the comparison baseline is called the control group.

Activity increased. Opening new repositories—spaces that store a project's code and change history—increased by 7.4%, and pushing fixed code up to the repository increased by 93.1%. The increase was larger on the side of repositories created from scratch by oneself (72.6%) than on the side of cloning other people's repositories (4.5%). However, as we see it, feedback loops could be mixed into the preceding two figures. This is because the design requires traces to remain in public commits to enter the treatment group, meaning people with increased public activity might be captured more easily. The abstract does not address this point separately.

Next comes the main body of this study. The authors tracked the newly created repositories using public security metrics to separate the two. One is security practices attached when first created, such as security policy documents or automated check settings. The other is the security status maintained by continuous upkeep, checking whether known vulnerabilities are currently absent. The practice side improved slightly and was similar regardless of experience. The security status worsened noticeably, and the steepness was greatest for beginners (DOI: 10.2139/ssrn.7032178, preprint). The abstract does not reveal how free this security comparison is from the aforementioned feedback loop.

The authors argue as follows: agentic coding breaks down the cost of exploration to try new things, but it does not substitute for the judgment required to sustain that exploration, and beginners, whose exploration increased the most, may possess that judgment the least. Judgment itself is not a value measured by this study.

Numbers That Split Depending on What You Count

There is a classification program (classifier) that picks out AI coding tool users by looking only at public commit records. It was trained on 74 confirmed users and 202 controls, for a total of 276 accounts, and the classification performance measured in cross-validation was 0.94. This score means that the closer it is to a perfect score, the better it distinguishes between users and non-users. When the same classifier was run on two analyses, the results told different stories.

When comparing before and after adoption at the account level against the control group, the number of commits per week increased significantly and the interval between commits decreased. However, in data (panels) tracking 34 countries grouped at the country level from 2022 to 2024, a stable negative relationship emerged between tool penetration and the number of commits per developer (weighted coefficient −7.56, p=0.05). This means that the higher the penetration rate, the fewer commits per person there were.

A pull request is a request to merge code you have fixed into a project. Multiple commits are sometimes bundled into a single pull request. In the same panel, pull requests per developer were estimated narrowly as showing no change (coefficient +1.33, p=0.76) (DOI: 10.2139/ssrn.6677399, preprint).

The authors interpret this divergence cautiously. It aligns with the explanation that rather than actual output decreasing, the tool changed commits into smaller numbers of larger chunks—in other words, changed the grain size of commits—but they note that their data cannot rule out alternative explanations.

There is a place for caution on our end as well. The two results just laid out were not measured with the same yardstick. The former is a before-and-after comparison at the account level, while the latter is a country-level panel, so both the unit of aggregation and the analytical design differ. The place where the authors say the results split depending on the scale is narrower. It is whether one counts commits or pull requests within the same panel. What this paper supports goes up to that point, and that alone reveals that simply changing what is counted in the same data can yield different conclusions.

Only a portion of what developers actually do is left as a record in the system, and a significant portion remains outside the data (DOI: 10.53606/evfu.25.620-628). This review, which surveys productivity measurement methods, calls this "invisible work," and adding the effect where metrics alter human behavior, it concludes that productivity cannot be measured with a single numerical metric.

The picture of the entire field lies in a systematic review of 39 peer-reviewed papers published from January 2014 to December 2024. Peer review is a process where researchers in the same field review a manuscript before publication. The first conclusion is that a majority report substantial gains, and the second is that a noticeable minority point out serious risks. Frequently reported gains are faster development speed, less time spent searching for code, and the automation of repetitive tasks. Frequently reported concerns are the habit of offloading thinking tasks to the tool and a decrease in team collaboration.

The third conclusion forms the backbone of this article. Here, the assistant is a language model—specifically, a model that takes text and outputs what should come next ("How Generative AI Answers") implemented as a tool. Whether a language model assistant raises or lowers code quality remains unresolved, and existing studies report contradictory results depending on context and evaluation criteria (DOI: 10.1145/3809494). The same review also points out biases in research design. 59% were exploratory studies, and there is a scarcity of studies measuring over a long period or at the team level.

Where Perception and Measurement Diverge

In an agile environment—a development method where work is broken into short cycles and released frequently—91 practitioners who use AI coding tools were asked. The item "work finishes quickly" scored the highest at an average of 4.37, followed by coding efficiency, organizing requirements, and the progress of short work cycles (sprints). Yet the same people worried just as intensely. "Opportunities to learn decrease" scored 4.26, over-reliance 3.79, and security issues 3.78 (DOI: 10.2139/ssrn.7331699, self-reported survey · sample of 91 · no measurement).

The same people rate "it got faster" and "I learn less" high at the same time. The three numbers that opened this article were a case of measuring this mismatch with a clock. Even after completing 246 tasks, the developers estimated they had become 20% faster, but in reality they had slowed down by 19%.

Mismatches between confidence and reality also exist on the AI side. That is what happens when you assign AI the task of swapping out an outdated electronic signature method for a new one designed to withstand even quantum computers. Two codebases—the entire body of code that makes up a program—two models, and four conditions were combined to run a total of 56 times. Even when the replacement part actually existed, successful runs were 0, and runs that accurately reported why they were blocked were also 0.

The details are more unsettling. When given the tools and a compiler—a program that converts human-written code into a form the computer can execute—failures did not decrease; they surfaced later. The sole compiled output reported success and claimed to use the new method, but the new method was not inside it. One model replaced it with the old method five out of five times, yet accurately stated the fact that the old method is vulnerable to quantum computers.

The same study reports on one intervention. When given the correct library—a bundled set of someone else's code you borrow and use—and how to call it, the swapping vanished. Still, the replacement failed, producing the same call error all five times, and the authors conclude with this. The failure here lies not in knowing which library to use, but in applying it (DOI: 10.20944/preprints202608.0599.v1, preprint). It is a narrow task with two codebases, but the picture is clear. Knowing and doing diverged, and they reported completion while diverged. The scene where the word "done" is attached to unfinished work is revisited in long tasks by "Multiplication Won't Tell You — Where Agents Break on Long Tasks".

Shifted Time: Review, Verification, and Maintenance

The 60-person experiment seen earlier reported that completion time decreased. Where that saved time went is the next question. Let's make it clear first: None of the studies below directly measured shifted time. What the studies below measured is where and how many flaws accumulate, and who cleans them up and when remains an unmeasured space.

Static analysis is an automated inspection that scans code without actually running the program to find suspicious spots. This is similar to a spell checker catching typos without reading the meaning. When AI-generated code was scanned on a large scale by attaching manual human review to three static analysis tools, this is what came out. Seven models were given the same problems in four languages with 100 per language to yield 2,800 code snippets, and vulnerabilities were found in 936 of them. There were 1,703 vulnerabilities in total, spanning 36 types. By language, they were uneven: 695 in JavaScript, 398 in Python, 348 in Java, and 262 in C++ (DOI: 10.2139/ssrn.6968361, preprint). Every model tested produced code containing vulnerabilities.

What is more concerning is what happens the more you fix it. While tracking changes in vulnerabilities as code was refined through conversations with multiple models to create 1,250 versions, vulnerabilities increased statistically significantly with each repetition of the refinement cycle. These were common types, such as cases where authentication was improperly configured or credentials like passwords were hardcoded into the code, and appeared across all evaluated model families. There are two additional findings. Verification that actually runs the program and testing that injects random inputs found more problems that static analysis alone missed. And a procedure incorporating security-conscious prompts and static analysis actually reduced the margin of increase. A prompt is the text put in front of a model to get an answer, with the exact meaning defined in "How Guessing the Next Word Becomes a Conversation". One more condition is attached to the authors' conclusion sentence: while functions are improving, security getting slightly worse by a measurable amount—in other words, security drift—can occur (DOI: 10.2139/ssrn.6958668, preprint).

Test code stands in the same place. A unit test is a short piece of inspection code that automatically checks whether a small piece of code works properly. When 194 Python unit tests created by a code completion tool were examined, 47.4% had one or more traces of bad habits that make later modifications difficult. The most common was a lack of description for what was being verified. The tests themselves were valid, but quality violations were frequently mixed in. This judgment was not left only to automated inspection, but went through evaluations by developers and quality personnel (DOI: 10.5753/sbes.2024.3561).

Technical debt is the cost that is hastily put together to run right now, but attaches more like interest when you modify it later. It is like postponing a credit card bill; the moment you postpone it, nothing happens.

A mixed methods study that actually measured this debt combined a survey of 412 experts, an analysis of 1,240 commits, and 22 in-depth interviews. Short-term productivity definitely rose ($\beta=0.48, p<.001$), especially in repetitive tasks. However, for experienced developers, when the burden of maintenance was also factored in, that gain was not statistically significant ($\beta=0.06, p>.05$).

The number in this study that is easiest to cite comes with conditions. Governance is a management system that determines who reviews and approves code under what rules. When good governance was absent, AI-assisted code had 37% higher debt per line of code and 41% higher security defects per line of code than human-written code. How well-established governance was was the core condition that governed this relationship ($\Delta R^2=0.14$). If you strip away the conditions, this number becomes a completely different claim. The authors propose a model managed across three axes—speed, maintainability, and security—and write that technical debt should be re-examined not as something that accumulates automatically, but as a risk tangled up with technology, people, and organizations (DOI: 10.70917/ijcisim-2026-5434).

A literature review examining 31 formal literature papers and 73 non-peer-reviewed literature items like blogs and industry reports—104 papers in total—gives this debt a new name. Fast-integration debt is a cascade that begins when hastily generated code prioritizes speed over quality, breeding management process debt and ultimately driving up long-term maintenance costs. The same review also organizes new categories like prompt, data, and provenance debt, and cites keeping humans in the loop, prompt design, and data quality alignment as countermeasures gleaned from the literature. At the end, it points out the blank spots: that there are still no standardized benchmarks nor metrics tailored to language models (DOI: 10.1145/3820165). Benchmarks are a collection of test problems set up to make multiple AI models solve the same problems and compare their scores, as described in "How AI Gets Things Wrong". To point out its grade, this review was published in a peer-reviewed venue, but 70% of its data consists of non-peer-reviewed literature.

A study that pre-emptively depicted what happens when technical debt accumulates is not an observation, but a model. Based on engineer interviews and literature, a simulation model was built to mimic the process changing over time, and the result was an inverted U-shape. There is definitely a section where AI assistance improves overall performance, but once that threshold is crossed, the accumulated debt eats away at maintainability and profitability. It also varied depending on conditions. Greenfields and fast-changing environments endured higher AI reliance, whereas legacy system environments and low-proficiency teams were much more vulnerable to long-term decline. The authors present paying down debt as a safety guardrail (DOI: 10.2139/ssrn.5842302, preprint). What the model tells us is what shape of curve to expect, not how bad it gets.

Same Team, Same Tools, Two Procedures

We write the rating before the numbers. This case is a preprint where a two-person team tracked and evaluated their own 12 months (DOI: 10.2139/ssrn.7114678, preprint · self-reported by a 2-person team). The sample is a single team, the evaluators are the parties involved, and there was no peer review. The numbers below should be read only as the shape of a single case, not as precise measurements.

Over 12 months, that team produced 2.6 million lines of code across 2,929 commits. The same team used the same tools under two different procedures, and the contrast between those procedures mirrors this article's question of comparing speed against what is left behind.

The side built without a procedure was 4.2 times faster by code volume. That is 360,334 lines per month versus 84,802 lines. A working demo with a prospective customer account attached even came out of it. However, they self-evaluated that they met only 2 of 18 enterprise audit items set by the authors, and the abstract notes alongside this an instance where passwords were saved in plain text without encryption. There were 0 test files, 0 deployment pipelines to automatically check and ship code, and 4 abandoned repositories. The authors' verdict is that the software cannot be sold to regulated customers and must be thrown away rather than fixed and reused. Counted by the number of features fit for a business, it was zero.

The side with a structured procedure attached was slower at one-quarter the volume, but it satisfied all 18 audit items and left behind 68 test files and 13 deployment pipelines. As for the proportion of created code that was discarded, it was 42% for the no-procedure side and 55% for the structured side, meaning the structured side threw away more.

One number in the abstract does not add up when you calculate it. The beginning of the abstract states that only 250,000 lines remained out of 2.6 million, putting the discard rate at 90%. Yet when calculated by combining the 55% and 42% rates per condition, the discard rate is 49%, and the remaining lines are 1.34 million. The two values do not share the same denominator. The abstract also does not reveal which condition came first. Therefore, what can be read from this case is only the contrast between the two conditions, and a story of improvement over time cannot be read.

It is worth pausing at the fact that the side which fulfilled all the audit items threw away more. When generation becomes cheap, the act of discarding becomes work in itself. That work is not visible if you only count the volume or speed of the code produced. This case only revealed it after separately counting the number of discarded lines.

Items Billed Later — Skill

From here on, the nature of the evidence changes. The paragraph immediately below is an argument and synthesis, not direct measurement.

A synthesis paper combining empirical studies since 2020 with educational theory notes that relying too heavily on AI assistance can weaken deep debugging skills—the ability to find and fix errors in a program—and hinder conceptual understanding (DOI: 10.62762/jse.2025.847963, synthesis paper · not a primary measurement). This paper's prescription is order: doing exploration with AI, walking through problem-solving and algorithmic thinking by hand once, and then bringing AI back in during the refinement stage. A computer society venue looks further ahead. The argument is that if generative AI—artificial intelligence that creates new content like text or images ("How Generative AI Answers")—takes over the effort of thinking at school while companies hire fewer juniors, the very pipeline that nurtures senior developers is threatened (DOI: 10.1145/3837767, essay venue).

We have pointed out that both claims are weak as evidence. However, two measured results from the earlier sections point to the same group. In the 60-person controlled experiment, the group that gained the most was juniors (DOI: 10.2139/ssrn.6932038, preprint), and in the 61,056-pair matched study, the group whose security posture worsened the most steeply was also novices (DOI: 10.2139/ssrn.7032178, preprint). The two studies used different data to measure different things. "Juniors" is a job grade and "novices" was split by public activity history, so they are not even groups defined by the same criteria. Still, the fact that gains and losses were both largest in the strata called by the same name is worth reading as a warning: if you decide where to deploy a tool by looking only at the gains, you might miss the losses.

Questions of the same shape have arisen outside of software. "Does More Automation Mean Fewer Human Errors?" asks about the place where humans monitor, while this article asks about the place where humans create. What overlaps is the shape of the question, so we do not import the conclusions from over there. If "The Score Went Up. What Went Up?" asked what went up when scores rose, what we asked here is what shrank when speed rose.

Five Questions to Ask

Looking at 2,547 conversations between developers and chatbots publicly shared on GitHub between May 2023 and June 2024, the conversations are short, task-oriented, and mostly end within 1 to 3 turns, meaning questions and answers go back and forth three times or fewer (DOI: 10.1145/3798163). This means it was common to handle one task briefly per conversation and be done. However, only conversations that people chose to share were collected, and that period precedes the generation of agent-like tools we saw earlier.

Therefore, "Do AI coding tools make developers faster?" is not a question that can be answered in a single sentence. A review summarizing 39 papers noted that while many report gains, whether they raise or lower code quality varies depending on the context and evaluation criteria, making it unresolved (DOI: 10.1145/3809494). What can be answered is a list of questions.

① What is being counted? Is it commits, projects, or actual releases? In the same data, the growth rate of autonomous coding agents diverged at 180%, 50%, and 30 for commits, projects, and releases respectively (DOI: 10.2139/ssrn.6859839, preprint). The moment you choose what to count, half of the answer is decided.

② Where did the saved time go? Is it review, rewriting, or maintenance? Where the human bottleneck lies has not yet been pinned down (DOI: 10.2139/ssrn.6859839, preprint). There is also a summary stating that quickly generated code increases long-term maintenance costs (DOI: 10.1145/3820165).

③ Who is using it? The stratum with the greatest gains and the stratum with the greatest risks may overlap (DOI: 10.2139/ssrn.7032178, preprint).

④ What kind of codebase is it? Is it a greenfield project or a legacy system? Model research says the two can withstand different levels of AI dependence (DOI: 10.2139/ssrn.5842302, preprint · simulation).

⑤ What is attached alongside it? There were no numbers without conditions attached. The debt and defect density of AI-assisted code were significantly higher than human-written code when good governance was absent (DOI: 10.70917/ijcisim-2026-5434), and the increase in vulnerabilities shrank when security-conscious procedures were inserted (DOI: 10.2139/ssrn.6958668, preprint).

This is no reason to lean into cynicism either. There is a controlled experiment reporting about a 21% reduction in enterprise-grade tasks (arXiv:2410.12944, preprint · internal research · large confidence interval). There are experiments where completion time, errors, and mental load went down together (DOI: 10.2139/ssrn.6932038, preprint), and the inverted U actually has a region where gains remain (DOI: 10.2139/ssrn.5842302, preprint · simulation). The increase in security vulnerabilities also shrank when procedures were inserted. Nor is there any reason to lean into optimism. The same studies all attached conditions to their final paragraphs, and those conditions were mostly that humans must keep watching.

One thing remains. Speed shows up immediately, but the bill may arrive later.

Code coming out quickly is felt this afternoon, and the time spent reading, fixing, and deleting that code may come later. That is the shape of the inverted U drawn by model research, and studies that track this time lag over a long period and measure it directly are still rare. That is also why the word debt fits here. Nothing happens the moment you borrow. If you set the measurement period shorter than that time lag, you might end up measuring only the gains.

Source Papers

Preprints are unpeer-reviewed public manuscripts.

• "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" (2025, preprint) — arXiv:2507.09089 — Randomized controlled experiment. 16 developers with some AI experience, 246 tasks, mature projects with an average of 5 years of experience, tools from February to June 2025. Pre-experiment prediction −24%, post-experiment self-estimation −20%, measured +19%. Expert predictions were reductions of 39% and 38%. After checking 20 conditions, concluded that while artifacts cannot be ruled out, it is unlikely that the slowdown was due to design.
• "How much does AI impact development speed? An enterprise-based RCT" (2024, preprint · Google internal research) — arXiv:2410.12944 — 96 Google full-time engineers, enterprise-grade tasks, 3 AI features, summer 2024. Significantly reduced task time, best estimate roughly 21% but with a large confidence interval. Developers with longer coding activity times were faster. Authors explicitly note that extrapolation is not possible.
• "Skill-Augmented AI Coding Agents: A Two-Layer Framework for SKILL.md Design" (2026) — DOI: 10.58806/ijmir.2026.v3i6n03 — Summary that controlled studies report 21–56% improvements alongside up to 19% delays (not direct measurement). Self-improvement claims are explicitly noted as self-reported.
• "An Empirical Evaluation of AI-Assisted Code Generation on Productivity and Code Quality" (2026, preprint) — DOI: 10.2139/ssrn.6932038 — 60 people, 8 tasks, 2 conditions, 960 observations. Reductions in completion time, errors, complexity, and cognitive load, with the largest gains in juniors. Also reports redundant abstraction, verbosity, and the need for additional verification.
• "Empirical Analysis of AI-Assisted Code Generation Tools: Quality, Security, Productivity" (2025, small multidisciplinary journal) — DOI: 10.36948/ijfmr.2025.v07i06.61350 — 120 people, 480 modules. Productivity +31.4%, maintainability +18.2%, security vulnerabilities +23.7%. Python +26.3%, C++ +34.8%.
• "Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools" (2026, preprint) — DOI: 10.2139/ssrn.6859839 — Over 100,000 GitHub developers + matched usage records. Commits 40/140/180%, projects 50%, releases 30% (meaning growth slowed, not that it decreased). Elasticity of substitution 0.25. New apps increased across four app stores, but total usage did not increase. Author conclusion is "only partially".
• "Agentic Coding, Developer Exploration, and Software Security" (2026, preprint) — DOI: 10.2139/ssrn.7032178 — 61,056 treatment group identified by public commit trailers vs. 61,056 matched control group. Repository creation +7.4%, pushes +93.1%, originals +72.6% vs. forks +4.5%. Minor improvements in security practices, state is declining and steepest among novices.
• "Detecting AI Coding Tool Adoption and Its Effects on Developer Commit Activity" (2026, preprint) — DOI: 10.2139/ssrn.6677399 — 276-account classifier (AUC 0.94). Account DiD and 34-country panel are separate analyses. Within the panel, a precise estimated null effect of −7.56 commits per person (p=0.05) and +1.33 pull requests (p=0.76). Authors explicitly note that alternatives cannot be ruled out.
• "Measuring Software Developer Productivity: Review of Approaches and Their Limitations" (2026) — DOI: 10.53606/evfu.25.620-628 — "Invisible work", the effect of metrics changing behavior. Productivity cannot be measured by a single metric.
• "The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review" (2026) — DOI: 10.1145/3809494 — 39 peer-reviewed papers (2014.1–2024.12). Many show gains, a few show severe risks. Code quality is unresolved and conflicts depending on context and criteria. Lack of longitudinal and team evaluations.
• "The Impact of AI-Assisted Coding Tools on Agile Software Development Practices" (2026, self-report survey) — DOI: 10.2139/ssrn.7331699 — 91 practitioners. Fastest completion highest at 4.37, reduced learning opportunities (4.26) and overreliance (3.79) also high.
• "Agentic Cryptographic Debt: Post-Quantum Migration Regression Under Autonomous AI Development" (2026, preprint) — DOI: 10.20944/preprints202608.0599.v1 — 56 runs. 0 successes even when drop-in replacement parts existed, and 0 runs accurately reported the reason for being blocked. The single compiled output reported success and advertised a new method, but it was absent. Once the correct library and calling convention were pointed out, the substitution disappeared, and the remaining failures lay in application rather than knowledge.
• "A Comparative Security Evaluation of AI-Generated Code Across LLMs and Languages" (2026, preprint) — DOI: 10.2139/ssrn.6968361 — Vulnerabilities in 936 out of 7 models, 4 languages, and 2,800 snippets, 1,703 instances, 38 types (3 SAST types + manual review). JS 695, Python 398, Java 348, C++ 262.
• "Security Weaknesses in LLM-Generated Source Code: Iterative AI-Assisted Development" (2026, preprint) — DOI: 10.2139/ssrn.6958668 — 1,250 versions. Significant increase in vulnerabilities with each iterative refinement. Execution and fuzzing found additional instances missed by static analysis. Security-aware prompts mitigated the magnitude of the increase. The conclusion is that security drift occurs "even while functionality improves".
• "Detecting Test Smells in Python Test Code Generated by LLM" (2024) — DOI: 10.5753/sbes.2024.3561 — 47.4% of 194 Python unit tests showed one or more bad smell traits (the most common being the lack of descriptions for assertion points). Developer and quality evaluator assessments appended.
• "Generative AI and Technical Debt: Productivity Gains versus Code Quality and Security Risk" (2026) — DOI: 10.70917/ijcisim-2026-5434 — Survey 412 people, 1,240 commits, 22 interviews. Short-term productivity β=0.48 (p<.001), including maintenance burden in experienced developers β=0.06 (p>.05). Without good governance, debt is 37% higher and security flaws 41% higher (ΔR²=0.14). Proposes a speed-maintainability-security three-axis model and a sociotechnical risk framework.
• "Faster Code, Deeper Debt? A Multivocal Literature Review on Technical Debt in LLM-Assisted Development" (2026) — DOI: 10.1145/3820165 — 104 papers (31 formal, 73 outside review). Rapid integration debt and its cascades. Lack of standard benchmarks and language model metrics. Countermeasures include frameworks keeping humans in the loop, prompt design, and data quality alignment.
• "Beyond Productivity: The Hidden Costs of Generative AI in Software Development" (2025, preprint · simulation) — DOI: 10.2139/ssrn.5842302 — Inverted U-shape and sweet spot. Greenfield environments endure better, while legacy and low-skill teams are vulnerable. Debt repayment is a safeguard.
• "Process Discipline as the Isolating Variable in AI-Assisted Enterprise Development: A Natural Experiment" (2026, preprint · 2-person team self-report) — DOI: 10.2139/ssrn.7114678 — 12 months, 2,929 commits, 2.6 million lines. The unproctored condition had 4.2 times the volume (360,334 vs. 84,802 per month) but met only 2 out of 18 audit items, had 0 tests, and 0 regulatory compliance features. The structured condition met all 18 items and had 68 tests. ⚠️The 90% discard rate in the abstract and the 55%/42% per-condition figures have different denominators.
• "Is AI Code Generation Undermining Developers' Problem-Solving Skills?" (2026, synthesis) — DOI: 10.62762/jse.2025.847963 — Signals that excessive reliance can weaken deep debugging skills and conceptual understanding. Proposes a 3-step framework of exploration, manual solving, and verification.
• "Preparing Junior Software Developers for the AI Era" (2026, essay) — DOI: 10.1145/3837767 — Argument that the pipeline producing senior developers is threatened.
• "Unveiling the Role of ChatGPT in Software Development: Developer-ChatGPT Interactions on GitHub" (2026) — DOI: 10.1145/3798163 — 2,547 conversations publicly shared from 2023.5 to 2024.6. Short and task-oriented, mostly 1 to 3 turns.