Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford

This Stanford study on developer productivity found 0 correlation between developers assessment of their own productivity and independent measures of their productivity. Any anecdotal evidence from developers on how AI has made them more or less productive is worthless. ▶ this video

As mentioned in the thread I linked, they acknowlege the productivity puzzle and try to control for it in their studies. It's worth reading them in detail, I feel like many of them did a decent job controlling for many factors. For instance, when measure the number of PRs they ensure that each one goes through the same review process whether AI-assisted or not, ensuring these PRs meet the same quality standards as humans. Furthermore, they did this as a randomly controlled trial comparing engineers without AI to those with AI (in most cases, the same ones over time!) which does control for a lot of the issues with using PRs in isolation as a holistic view of productivity. >... whose participants are working for organizations that are heavily invested in future success of AI. That seems pretty ad hom, unless you want to claim they are faking the data. Along with co-authors who are from premier institutes like NBER, MIT, UPenn, Princeton, etc. And here's the kicker: they all converge on a similar range of productivity boost, such as the Stanford study: > ▶ this video (from Stanford, not an RCT, but the largest scale with actual commits from 100K developers across 600+ companies, and tries to account for reworking AI output. Same guys behind the "ghost engineers" story. The preponderence of evidence paints a very clear picture. The alternative hypothesis is that ALL these institutes and companies are colluding. Occam's razor and all that.

The folks at Stanford in this video have a somewhat similar dataset, and they account for "code churn" i.e. reworking AI output: ▶ this video -- I think they do so by tracking if the same lines of code are changed in subsequent commits. Maybe something to consider.

"That study design has some issues. " This is a study that tries to be scientific, unlike developer self reports and CEO promises of 10x. Can you point to a better study on the impact of AI on developer productivity? The only other one I can think of finds a 20% uplift in productivity. ▶ this video

This contains some specific data with pretty graphs: ▶ this video But if you do professional development and use something like Claude Code (the current standard, IMO) you'll quickly get a handle on what it's good at and what it isn't. I think it took me about 3-4 weeks of working with it at an overall 0x gain to realize what it's going to help me with and what it will make take longer.

> The problem is, there are very few if any other studies. Not at all, the METR study just got a ton of attention. There are tons out there at much larger scales, almost all of them showing significant productivity boosts for various measures of "productivity". If you stick to the standard of "Randomly controlled trials on real-world tasks" here are a few: link (4867 developers across 3 large companies including Microsoft, measuring closed PRs) link (1219 programmers at a Chinese BigTech, measuring LoC) ▶ this video (from Stanford, not an RCT, but the largest scale with actual commits from 100K developers across 600+ companies, and tries to account for reworking AI output. Same guys behind the "ghost engineers" story.) If you look beyond real-world tasks and consider things like standardized tasks, there are a few more: link (96 Google engineers, but same "enterprise grade" task rather than different tasks.) link (25 professional developers across 7 tasks at a Finnish technology consultancy.) They all find productivity boosts in the 15 - 30% range -- with a ton of nuance, of course. If you look beyond these at things like open source commits, code reviews, developer surveys etc. you'll find even more evidence of positive impacts from AI.

Most of those studies call this out and try to control for it (edit: "it" here being the usual limitations of LoC and PRs as measures of productivity) where possible. But to your point, no, there is still a strong net positive effect: > ▶ this video (from Stanford, not an RCT, but the largest scale with actual commits from 100K developers across 600+ companies, and tries to account for reworking AI output . Same guys behind the "ghost engineers" story.) Emphasis added. They modeled a way to detect when AI output is being reworked, and still find a 15-20% increase in throughput. Specific timestamp: ▶ this video

Thank you! > ▶ this video (from Stanford, not an RCT, but the largest scale with actual commits from 100K developers across 600+ companies, and tries to account for reworking AI output. Same guys behind the "ghost engineers" story.) I like this one a lot, though I just skimmed through it. At 11:58 they talk about what many find correlates with their personal experience. It talks about easy vs complex in greenfield vs brownfield. > They all find productivity boosts in the 15 - 30% range -- with a ton of nuance, of course. Or 5-30% with "Ai is likely to reduce productivity in high complexity tasks" ;) But yeah, a ton nuance is needed

So far I've found that the people who are hating on AI are stuck maintaining highly coupled that they've invested a significant amount of mental energy internalizing. AI is bad on that type of code, and since they've invested so much energy on understanding the code, it ends up taking longer for them to load context and guide the AI than to just do the work. Their code base is hot coupled garbage, and rather than accept that the tools aren't working because of their own lack of architectural rigor, they just shit on the tools. This is part of the reason that that study of open source maintainers using Cursor didn't consistently produce improvement (also, Cursor is pretty mid). ▶ this video is one of the largest studies I've seen so far and it shows that when the codebase is small or engineered for AI use, >20% productivity improvements are normal.

I'm not the OP and I"m not saying you are wrong , but I am going to point out that the data doesn't necessarily back up significant productivity improvements with LLMs. In this video ( link ) they present a slide by the company DX that surveyed 38,880 developers across 184 organizations, and found the surveyed developers claiming a 4 hour average time savings per developer per week. So all of these LLM workflows are only making the average developer 10% more productive in a given work week, with a bunch of developers getting less. Few developers are attaining productivity higher than that. In this video by stanford researchers actively researching productivity using github commit data for private and public repositories ( ▶ this video ) they have a few very important data points in there: 1. There's zero correlation they've found between how productive respondants claim their productivity is and how it's actually measured, meaning people are poor judges of their own productivity numbers. This does refute the claims on the previous point I made but only if you assume people are wildly more productive then they claim on average. 2. They have been able to measure actual increase in rework and refactoring commits in the repositories measured as AI tools become more in use in those organizations. So even with being able to ship things faster, they are observing increase number of pull requests that need to fix those previous pushes. 3. They have measured that greenfield low complexit

"How do we reconcile these two comments? I think that's a core question of the industry right now." There is no correlation between developers self assessment of their productivity and their actual productivity. ▶ this video

The studies I've seen for AI actually improving productivity are a lot more modest than what the hype would have you believe. For example: ▶ this video Skepticism isn't the same thing as fighting the future. I will call something AGI when it can reliably solve novel problems it hasn't been pre-trained on. That's my goal post and I haven't moved it.