AI's elusive coding speedup

Driving the news: The study by METR, a nonprofit independent research outfit, looked at experienced programmers working on large, established open-source projects.

• It found that these developers believed that using AI tools helped them perform 20% faster — but they actually worked 19% slower.
• The study appears rigorous and well-designed, but it's small (only 16 programmers participated, completing 246 tasks).

Zoom out: For decades, industry visionaries have dreamed of a holy grail called "natural language programming" that would allow people to instruct computers using everyday speech, without needing to write code.

• As large language models' coding prowess became evident, it appeared this milestone had been achieved.
• "The hottest new programming language is English," declared AI guru (and OpenAI cofounder) Andrej Karpathy on X early in 2023, soon after ChatGPT's launch.
• In February, Karpathy also coined the term "vibe coding" — meaning the quick creation of rough-code prototypes for new projects by just telling your favorite AI to whip up something from scratch.
• The most fervent believers in software's AI-written future say that human beings will do less and less programming, and engineers will turn into some combination of project manager, specifications-refiner and quality-checker.
• Either that, or they'll be unemployed.

Zoom in: AI-driven coding tends to be more valuable in building new systems from the ground up than in extending or refining existing systems, particularly when they're big.

• While innovative new products get the biggest buzz and make the largest fortunes, the bulk of software work in most industries consists of more mundane maintenance labor.
• Anything that makes such work more efficient could save enormous amounts of time and money.

Yes, but: This is where the METR study found AI actually slowed experienced programmers down.

• One key factor was that human developers found AI-generated code unreliable and ended up devoting extra time to reviewing, testing and fixing it.
• "One developer notes that he 'wasted at least an hour first trying to [solve a specific issue] with AI' before eventually reverting all code changes and just implementing it without AI assistance," the study says.

Between the lines: The study authors note that AI coding tools are improving at a rapid enough rate that their findings could soon be obsolete.

• They also warn against generalizing too broadly from their findings and note the many counter-examples of organizations and projects that have made productivity gains with coding tools.

One notable caution that's inescapable from the study's findings: Don't trust self-reporting of productivity outcomes. We're not always the best judges of our own efficiency.

• Another is that it's relatively easy to measure productivity in terms of "task completion" but very hard to assess total added value in software-making.
• Thousands of completed tickets can be meaningless — if, for instance, a program is about to be discontinued. Meanwhile, one big new insight can change everything in ways no productivity metric can capture.

The big picture: The software community is divided over whether to view the advent of AI coding with excitement or dread.

• "Quitting programming as a career right now because of LLMs would be like quitting carpentry as a career thanks to the invention of the table saw," the AI developer and blogger Simon Willison recently wrote on Bluesky.
• Or "it could be like quitting agriculture as a career thanks to the invention of tractors and combine harvesters instead," industry veteran Tom Coates replied.