Generative AI can write lots of code but its overall impact on developer productivity is more nuanced. Research from Atlassian’s DX, Google, LaunchDarkly and others, shows high variation between companies, and well-known factors in achieving high performance – relevant long before AI tools emerged – still apply today.

Justin Reock, deputy CEO at DX – a developer intelligence platform acquired by Atlassian which helps to assess developer productivity – presented anonymized, aggregated data drawn from upwards of 135,000 developers at 400 DX customers, dating from mid-October this year, at the AWS re:Invent conference last week. 

A Google DORA (DevOps Research and Assessment) report earlier this year found modest gains in code quality from GenAI adoption, measured as 2.6 percent on average, but only a 0.11 percent gain in change failure rate, which is how often a feature has to be rolled back because it broke something.

According to Reock, this research is misleading, because it is an average that disguises huge variations across different companies. In other words, some companies do much better than others, with change confidence increased by 20 percent or more, while others do much worse and suffer a high negative impact.

The key to doing well with AI, said Reock, is to do a “lot of work on SDLC [software development lifecycle] and getting all the right code hygiene things” – which turn out to be the same factors which applied before AI coding came along. AI, he said, “might finally get companies unified about caring about the things we should have been doing for developers years ago.”

A similar volatility applies to code maintainability, with the average gain of 2 percent hiding the fact that some teams achieve much better maintainability with AI, others much worse.

Simply using AI code completion saves developers around 3.8 hours a week, according to DX metrics: more significant than it appears because most developers only spend a minority of their time writing code. Time savings from AI though are small compared to pain points that “have nothing to do with the generation of code,” said Reock, such as meeting-heavy days, frequent interruptions, build times, review waiting time, and other factors.

Reock presented the ten top use cases for AI assisted code, based on research in its guide to AI-assisted engineering, with the top use case being stack trace analysis.

AI users ship 60 percent more pull requests (proposed code changes) according to DX, though this statistic could be positive or negative depending on whether those code changes were valuable or time-wasting AI slop.

A new report from LaunchDarkly, based on a survey of 767 engineers and DevOps leaders, found that 94 percent reported accelerated coding thanks to AI, but 91 percent had low trust in shipping AI-developed code to production. 81 percent of teams ship code with unresolved risks, citing pressure to deliver new features on time, and resulting in production incidents.

Another trend DX identified is that junior developers use AI the most, with 41.3 percent daily usage versus 32.7 percent for a staff engineer; yet the more senior developers save more time, something Reock attributed to their greater efficiency and ability to judge when code is well written.

The programming language used also makes a difference. Go developers can expect to save 4 hours a week from use of AI, whereas COBOL programmers achieve only 2 hours. Reock said this is because AI works better with “modern languages”.

There is a J curve when companies first adopt AI for software development, Reock said, with initial productivity and quality decline, but improvements later on as adoption increases.

Assessing the impact of AI is complicated by the fact that more non-developers or occasional developers such as designers and engineering managers are shipping code as a result of AI, not necessarily production code but things like working prototypes. “We have to start rethininking who is a developer,” said Reock. He sees this as a positive because somebody who is able to contribute code to a project will have more empathy with the core development team.

Another question is whether AI is impeding developer learning, because coders simply accept the AI-generated code without understanding it. That is a choice, said Reock. Those who want to become better developers will take the time to study the code and learn what it does; those who do not will skip that step, just as was the case with copy and paste from Stack Overflow in the past.

Creating a development environment which encourages flow state and minimizes context switching has huge impact on productivity, outweighing the impact of AI. “AI is great for many things, but not a silver bullet,” said Reock.

What is the most important factor in productivity with AI? According to Reock, it remains the same as before AI coding, a factor he called psychological safety within a team. “That’s more important now than ever, with AI,” he told re:Invent attendees.