Limited Impact and Integration: The Reality of GenAI in Software Development

Introduction

In recent years, Generative AI (GenAI) has gripped the imaginations of technologists and business leaders alike. It promises to transform the software development industry through enhanced productivity and innovation. Popular opinion suggests that GenAI is already deeply embedded in software development and achieving unprecedented results.

McKinsey estimates that GenAI could lift productivity in software engineering globally by 31% (equivalent to $485 billion), in the corporate IT sector alone. 

Given these optimistic figures, it is unsurprising that enthusiasm and expectations for GenAI are so high.

This is reflected in KPMG’s 2023 survey of business leaders, of whom 78% viewed GenAI as the top emerging technology over the next three to five years. 

However, a closer look at the impact of GenAI in code committed to version control systems in enterprise software development reveals a significant gap between the hype around its potential and the reality of its current usage. 

Drawing on insights from BlueOptima’s unprecedented study, this article challenges the prevailing narrative

Understanding the Gap

BlueOptima’s report, The Impact of Generative AI on Software Developer Performance, is the largest and most comprehensive of its kind to date. The study draws on an analysis of over 218,000 developers working in an enterprise setting. Its findings paint a picture of GenAI as having high potential, but not yet high impact.

Only 2,062 of this study group consistently contributed GenAI-authored code that the developers themselves did not significantly rework. This represents less than 1% of the analysed developer population demonstrating real confidence in allowing GenAI code to go into production without thorough human supervision.

Even more strikingly, the study reveals that, of over 3,000 developers given licenses to GenAI tools, only 12% committed GenAI-authored code multiple times without making significant modifications.This demonstrates that, even in the context of an engineering team in which AI is being actively promoted from the top down, uptake and usage of licensed tools is far from popular.

These findings suggest that GenAI usage and integration into the developer workflow could be better. 

BlueOptima’s research corroborates findings from other studies. For example, Boston Consulting Group’s December 2023 survey of industry software leaders found that only 30% of the participating companies had adopted GenAI for software development. Moreover, across roughly 75% of these early adopters, less than half of developers were using the technology such that they were able to commit the code produced by GenAI without material changes.

The limited usage of  GenAI tools among developers is striking, especially considering the aggressive marketing and the high expectations set by AI vendors. 

Industry experts have often touted AI as a revolutionary force in software development. Yet, the reality is markedly different, with many developers continuing to rely on traditional methods. This discrepancy raises questions about both the technology and its users’ readiness to fully commit to it in their workflows.

Challenges to AI usage

Many issues can often profoundly hinder the complete integration of GenAI in software development.

Complexity and usability issues with GenAI tools often mean a steep learning curve and a potential clash with established workflows, deterring developers from utilising them.

Cultural resistance also plays a significant role. Many developers need clarification about AI-generated code and are apprehensive about its reliability and maintainability.

In regulated industries, the integration of GenAI also faces hurdles related to security and compliance risks, especially if not properly vetted.

These are just some factors contributing to the lower integration rates of GenAI in software development environments.

Conclusion 

GenAI tools have the potential to improve development productivity while maintaining code quality.
However, progress in the integration of GenAI in the developer workflow is significantly more limited than often portrayed.

The low commit rate of GenAI-authored code without human rework highlights the need for a more agile, and more measured approach to its implementation, and more research into its long-term impact on engineering team dynamics.

Perhaps more importantly, it shows that the focus must remain on human developers, ensuring that the use of GenAI tools does not undermine their talent, productivity, or progress. This approach will safeguard the essential human elements of software development while exploring the benefits of GenAI. To explore more insights uncovered in BlueOptima’s study of GenAI usage amongst enterprise developer teams, read the full report here.