Amitai Schleier
@schmonz@schmonz.com
If your business makes software, I might be good for your business.
RE: https://mas.to/@carnage4life/117410940896619323
Wow, what an awesome research paper. Quality source data, fantastic analysis, great conclusions. It's long, but readable, and well worth your time.
I drew a few conclusions that I think they missed, but I'm going to wait to enumerate them until a second read-through.
Greg boostedHarvard researchers studied 718 companies and found AI coding agents increased lines of code by 30%, but no meaningful increase in completed tasks or projects.
Meanwhile, review time rose 49%.
Generating code faster hasn’t translated into completing more work. The bottleneck has moved to code reviews.
The solution the industry is landing on is throwing more AI at the problem. AI writes the code, AI reviews the code and no one knows how it works.
@narthur I have this on my iPad to read. 78 pages. Tried to start reading it on my flight from Saigon to Singapore on the weekend. Fell asleep unfortunately
Looking forward to your commentary for certain.
OK, I re-read this.
The source dataset is fantastic. 300MM events from 700K workers across 700 firms, across 5 years, with employment, PR (GitHub), issue (Jira), calendar, and AI usage integrated together.
The study design and analysis seem excellent. I saw no major gaps in their thinking or approach, nor flaws in their conclusions.
The graphs (which tell a lot of the story!) start on page 39.
Here's what *they* said:
LLM "assistants" (chat, IDE autocomplete, etc.) speed up programming - developers make about 9% more commits per unit time.
LLM "agents" (that do full tasks for you) make more code - 30% more LOC, 20% more commits, and 23% more PRs.
PR review time goes up 49% for agents, but 0% for assistants.
PRs where someone requested a code change go up nearly 100% for agents, but 0% for assistants.
Comments per PR go up 35% for agents, but 0% for assistants.
...even with AI PR review tools.
LLM agents increase code volume but reduce code quality, which shifts the work to the code review phase.
LLM usage of any sort does not produce a net speedup in "issues" or "epics" closed, per unit time. (And there's no evidence that the issues/epics got larger, because of LLMs.)
Nor do LLMs produce net staffing reductions, even to junior staff. (But there were some footnotes that are worth examining.)
Here are *my* thoughts:
Shifting the work to the code review phase is a disaster! That makes the work harder and less fun for developers, and leaves the team with less understanding of how the code works. And cycle time goes up, reducing actual business agility.
It almost certainly leads to lower code quality and higher complexity, causing a slow, ongoing growth in development friction that will eventually make the code unmaintainable by developers. Which leads to a high dependency on LLMs - and LLM vendors - until even the LLMs can't understand the code either. (Which already happens, even for human-generated code.)
So LLM assistants seem to be "development as we know it, but easier for the developers".
LLM agents seem to be a very expensive way to destroy your product.
LLM code review isn't really addressed, but it seems probably useful.
Overall, LLMs aren't actually providing higher product-development velocity because the bottleneck was never coding speed - it was "functional design speed," "testing speed," or "releasing speed". Those things haven't been improved (much) by LLMs.
And think of all the money spent on LLM agents!
@narthur net net : technical debt acceleration machine
@sumisu yep. A very expensive one.
(But not in “assistant” mode. Then they seem to be net zero. Or maybe beneficial in “make hard problems easy” and in PR review.)
I think LLMs are clearer wins in product-market fit spike-like scenarios, and then in the happy path they can also be pretty helpful for making valuable code that’s hard to change easier to change. But the temptations remain immediate.
1. No clear short-term savings from reduction in labor costs
2. Increased code volume and decreased quality correlate with a familiar upwards-bending cost curve… assuming the economics work the same for agent- or assistant-maintained code… assuming maintenance matters the way it used to
3. Making code review “load-bearing” is risky and invites failure
@schmonz @sumisu yes to all three.
On your economics question in point 2: there was another excellent writeup about “decoherence” recently, that I think answers it thoroughly: https://maevyn.codes/decoherence
And I think this week has been a remarkable demonstration of what happens when you let quality dip too far, in software systems! I don’t think we can really afford to just let things suck.
OK, I re-read this.
The source dataset is fantastic. 300MM events from 700K workers across 700 firms, across 5 years, with employment, PR (GitHub), issue (Jira), calendar, and AI usage integrated together.
The study design and analysis seem excellent. I saw no major gaps in their thinking or approach, nor flaws in their conclusions.
The graphs (which tell a lot of the story!) start on page 39.
Here's what *they* said:
LLM "assistants" (chat, IDE autocomplete, etc.) speed up programming - developers make about 9% more commits per unit time.
LLM "agents" (that do full tasks for you) make more code - 30% more LOC, 20% more commits, and 23% more PRs.
PR review time goes up 49% for agents, but 0% for assistants.
PRs where someone requested a code change go up nearly 100% for agents, but 0% for assistants.
Comments per PR go up 35% for agents, but 0% for assistants.
...even with AI PR review tools.
LLM agents increase code volume but reduce code quality, which shifts the work to the code review phase.
LLM usage of any sort does not produce a net speedup in "issues" or "epics" closed, per unit time. (And there's no evidence that the issues/epics got larger, because of LLMs.)
Nor do LLMs produce net staffing reductions, even to junior staff. (But there were some footnotes that are worth examining.)
Here are *my* thoughts:
Shifting the work to the code review phase is a disaster! That makes the work harder and less fun for developers, and leaves the team with less understanding of how the code works. And cycle time goes up, reducing actual business agility.
It almost certainly leads to lower code quality and higher complexity, causing a slow, ongoing growth in development friction that will eventually make the code unmaintainable by developers. Which leads to a high dependency on LLMs - and LLM vendors - until even the LLMs can't understand the code either. (Which already happens, even for human-generated code.)
So LLM assistants seem to be "development as we know it, but easier for the developers".
LLM agents seem to be a very expensive way to destroy your product.
LLM code review isn't really addressed, but it seems probably useful.
Overall, LLMs aren't actually providing higher product-development velocity because the bottleneck was never coding speed - it was "functional design speed," "testing speed," or "releasing speed". Those things haven't been improved (much) by LLMs.
And think of all the money spent on LLM agents!
@narthur net net : technical debt acceleration machine
@sumisu yep. A very expensive one.
(But not in “assistant” mode. Then they seem to be net zero. Or maybe beneficial in “make hard problems easy” and in PR review.)
I think LLMs are clearer wins in product-market fit spike-like scenarios, and then in the happy path they can also be pretty helpful for making valuable code that’s hard to change easier to change. But the temptations remain immediate.
1. No clear short-term savings from reduction in labor costs
2. Increased code volume and decreased quality correlate with a familiar upwards-bending cost curve… assuming the economics work the same for agent- or assistant-maintained code… assuming maintenance matters the way it used to
3. Making code review “load-bearing” is risky and invites failure
@schmonz @sumisu yes to all three.
On your economics question in point 2: there was another excellent writeup about “decoherence” recently, that I think answers it thoroughly: https://maevyn.codes/decoherence
And I think this week has been a remarkable demonstration of what happens when you let quality dip too far, in software systems! I don’t think we can really afford to just let things suck.
Faced with a problem, don’t just work to solve it. Instead, figure out what would have made it so easy that someone would have already solved it by now. Make it so. Then be the lucky someone.
NetBSD turns 33 this Sunday! 🚩
To celebrate 33 years of clean code, portability, and zero bloat, Challenging the rest of the fediverse to help hit this year's funding goals.
Also do drop a screenshot of your uptime, uname -a, or a pic of the weirdest hardware you've got running NetBSD right now. (RockPro64 NPF routers or Pi's hooked up to retro CRTs highly encouraged).
Throw some money at the developers keeping the real UNIX alive:
https://www.netbsd.org/donations/
#NetBSD #UNIX #RetroComputing #OpenSource #runbsd #FreeBSD #OpenBSD #Linux
1. Make full use of my particular set of skills
AND (this is a big and)
2. Have their results be appreciated.
Here’s what I’ll see when I occasionally go to the office.
Alt...
NYC skyline
Its replacement: a 1968 Baldwin Model R Artist Series my parents found for an even better bargain. Thanks, new friend, for all the music to come.
Alt...
A couple ivories missing
Alt...
MacBook Air 11” on a tray table
(& Medtner!)
Alt...
Haydn, Zappa, Wolpe, Oswald, Medtner, Rachmaninoff
The Leading Technical Change Seminar is *back*.
6 Attendees
4 2-hour Sessions
1 topic
Real Change In The Real World
April 6,7, 9 & 10, 1-3pm EST
LTC is a true seminar, led by me. Each attendee brings their own change challenges, and we tackle them all, as a group.
An alumnus writes: "A masterclass in leading without authority. Intensely practical and focused on outcomes, you leave with a toolkit that you can start applying immediately."
Grab your seat today!
https://www.geepawhill.org/courses/leading-technical-change/
Alt...
Donkey face on my shoulder
Booted 8’s ISO without any special grub flags. (On this box, whoa.) Easy repair.
These were my final big questions. I now trust this system with my data. Blog post soon.
Alt...
Please reboot the system, they said
Alt...
Debian 13 etc
1. Get fastfetch and all of its dependencies building on Mac OS 10.9 #MavericksForever
2. Include pkgsrc in the package report
Alt...
fastfetch output
That’s your clue: refactor first.
(Special case of the general KFB wisdom “First make the change easy, then make the easy change.”)
Hit up your network before applying. If you’re reading this, I’m in your network.
History
![[?]](https://schmonz.com/snac/schmonz/s/506a4e5be3c4eba4d0539c833aea957a.jpg)
![[?]](https://media.hachyderm.io/accounts/avatars/115/878/478/723/365/796/original/6c02f4cb72c0618d.png)
![[?]](https://media.mas.to/accounts/avatars/109/537/467/971/371/646/original/c0e4639a3b0935d0.jpg)
![[?]](https://cdn.famichiki.jp/accounts/avatars/116/922/237/227/071/743/original/ad0d11a4e680e7e3.jpeg)
![[?]](https://files.mastodon.social/accounts/avatars/000/031/175/original/bf44c90ea009d3cb.jpeg)
![[?]](https://media.bsd.network/accounts/avatars/000/006/949/original/9592eac6e5018e39.png)
![[?]](https://files.mastodon.social/accounts/avatars/108/193/376/617/364/330/original/f3fc0128b5118cb3.jpg)
![[?]](https://mastodon.sdf.org/system/accounts/avatars/108/233/134/134/174/252/original/75cf6c02b16edbb2.png)
