Search

Generative AI (GenAI) promises to revolutionize the future of business, including software development. Experts say GenAI advancements will drive developer productivity by up to 56%, and executives are taking notice. Bloomberg Intelligence finds that rising demand for generative AI products, including copilots that accelerate coding, could result in $280 billion of new software revenue by 2032. However, GenAI isn’t a fix-all for optimizing processes and boosting revenue. As with any technology, GenAI is an applied skill, requiring training and enablement for people and organizations to reach new levels of productivity and innovation. Business leaders must proceed with the right GenAI enablement to avoid adverse effects that can come with this next technological wave.

GenAI enables new ways to drive value through productivity, differentiation, and disruption. However, effectively implementing GenAI into your business isn’t as simple as just buying an enterprise software subscription or encouraging individual usage of one of the widely available generative pre-trained transformer (GPT) solutions that exist today. You need the right tools, the right training, consistent implementation across the development lifecycle, and executive buy-in to realize the true productivity uplift that GenAI can provide.

At Slalom, we started experimenting with GenAI in late 2022 to understand the possibilities, limitations, and best practices of this technology. Throughout 2023 we ran several internal initiatives to enable everyone across the organization with GenAI, including baseline training for all employees, a pilot program enabling technologists with GitHub Copilot, and Project 1k — an initiative to build and share 1,000 GenAI demos in 100 days. Culturally, these initiatives provided excitement and encouragement across teams. It also fostered collaboration across different areas of the business as strategists, designers, developers, and sales teams worked together to build, rapidly test, share, and iterate on GenAI solutions in real time.

Focusing in on developer productivity, we gained valuable insights into how GenAI is helping developers work better, faster, and more efficiently, as well as identified a set of actions needed to truly deliver value at scale. While initial development gains were impressive, we found that productivity fluctuates throughout the development lifecycle, both within sprints and across the development process from intake to testing.

For example, initial experiments showed impressive gains in early development sprints. According to Danny Weldon, director of software engineering at Slalom, in some cases we saw teams double sprint goals week over week. However, within sprints themselves we observed large fluctuations in commit history activity in GenAI usage. This prompted further research into the factors responsible for this phenomenon.

Looking across the development lifecycle more broadly, gains in one portion of the development lifecycle don’t necessarily translate to gains across the whole lifecycle, and in some cases may even create bottlenecks, slowing down the production engine. For example, if the engineering output skyrockets, exceeding testing capacity, then the testing phase becomes a bottleneck. Additionally, when new features are developed swiftly but the backlog is not adequately stocked, the development engine runs out of fuel, halting further advancements. Furthermore, if the pipelines—crucial to the efficient movement of features through the development lifecycle—fall behind, it impedes the deployment of features to the production environment. This interconnectedness underscores the need for a balanced and synchronized approach to GenAI deployment in software development, where each component supports and feeds into the other for optimal outcomes.

In addition to variability across the development lifecycle, GenAI productivity gains also vary by individual. While some engineers see large gains with the help of GenAI coding assistants, others see more modest increases, and some are even slowed down as they onboard to this new technology. Prompt engineering is a skill. Crafting clear, concise, and specific prompts enables the AI to generate more accurate and relevant code suggestions, ultimately enhancing productivity and code quality. Teams must therefore be trained and proficient in how they are using GenAI.

Understanding these fluctuations in productivity can be tricky but understanding is ultimately key to unlocking productivity gains with GenAI. As we’ve discussed, simply increasing in lines of code per engineer per day with the assistance of GenAI doesn’t guarantee an increase in story points completed. This can be due to bottlenecks created by speeding up only one part of the production cycle, or issues with the AI-generated code itself due to poor prompting leading to unproductive returns or code that doesn’t adhere to enterprise development and security standards. Operationalizing DevOps Research and Assessments (DORA) metrics before implementing GenAI can help teams benchmark productivity to better understand the impacts of GenAI, including how much faster code is deployed, how much more efficient the development process becomes, how quickly they can resolve issues, and how code quality improves, all of which can contribute to overall productivity gains.

By embracing experimentation and exploring areas of difficulty, Slalom has gained valuable insights into unlocking developer productivity that have shaped our strategy for helping teams harness GenAI.

We recently helped a leading health insurance provider enhance engineering productivity and developer experience through the implementation of GenAI coding assistants. By conducting a thorough evaluation and pilot program, including identifying use cases for pilot evaluation, upskilling developers on how to effectively use GenAI code generation tools, assessing developer productivity and code quality, and evaluating sentiments with key use cases and developer workflow with GenAI tools, we observed significant uplift across the development lifecycle. The pilot found an increase in productivity, standardization, and code quality across numerous coding tasks, particularly generating code and creating unit tests. Specific benefits included …

• Accelerated engineering: The overall results showed a 2.5x increase in code created, without a decrease in quality or adherence to standards.
• High efficiency and flow: 75% of developers reported increased efficiency and flow owing to less context switching and mental effort on non-value-add tasks.
• Improved quality: We saw a 34% increase in unit tests written and 83% of developers reported improved quality when performing quality checks.
• Faster modernization: 73% faster migration to modern programming languages and 3x faster migration to AWS Cloud Development Kit (CDK).
• Excellent career growth opportunities: Participants loved the skill enhancement components and hands-on support, leading to higher job satisfaction.

These successes have paved the way for broader implementation of GenAI tools across the organization, demonstrating the potential of these cutting-edge technologies to drive productivity and innovation.

As more and more organizations make investments in GenAI, it’s important to focus on the factors that will enable you to realize the greatest returns on your investment. Here’s a checklist to consider as you embark on your GenAI journey to help set your developers, teams, and business up for success:

1. Business analysis

• Identify ROI potential: Begin by researching tools and projected gains to estimate the potential time savings, increased productivity, or quality improvements GenAI can offer. Use this as a baseline to calculate the financial impact and justify the investment.
• Identify use cases: Find tasks that are repetitive, time-consuming, or where automation would reduce workloads for team members across the development lifecycle. Be sure to consider the full development lifecycle and consider up- and downstream impacts.
• Align with strategic goals: While it can feel like an imperative to start experimenting with AI, GenAI is ultimately an investment. If you cannot align your AI objectives with your strategic goals, revisit your objectives to determine initiatives that will drive operational efficiencies, enhance customer experiences, improve decision-making, and ultimately deliver measurable business value.

2. Feasibility

• Tool evaluation: Research available GenAI coding assistant tools. Consider popular products including Amazon CodeWhisperer, GitHub Copilot, Code Llama, and Google Gemini. Assess factors like cost, existing skill sets, and customizations that may be required for your organization.
• Integration complexity: Determine how and where you will be integrating tools into your development workflow. Will you be able to do this within your team, or will you need external expertise to design and set up the right workflows?

3. Readiness

• Team and organization: Adopting GenAI is a cultural shift requiring buy-in and enablement at every level. Leaders need to champion AI initiatives and provide the necessary resources and guidance. At the same time, teams must be excited and willing to adopt the technology. Connect them with other developers outside your organization to learn more about how these tools are making an impact.
• Governance and security: Define who will be able to access GenAI tools and implement ways to monitor access. Make sure you have a plan to address the security aspects of GenAI and that you are not training external models on proprietary code.

4. Investment

• Budget: Estimate the cost for the GenAI tools, integration of the tools, potential infrastructure upgrades, training, and standard maintenance of new technology.
• Training: Even, and in some cases especially, for the most experienced developer, the computing adage “garbage in, garbage out” applies to GenAI tools. As AI continues to advance, we will need to provide teams with additional training to make sure they are leveraging the best the technology has to offer.
• Pilot: GenAI use cases are best to start in a carefully planned pilot. You want to control the environment, measure impact, and iterate on your approach. Here’s where DORA metrics can be especially helpful in measuring productivity gains.
• Pilot exit strategy: Develop a plan to roll it out more broadly across the organization once the pilot is complete or decide that GenAI did not provide the intended results to achieve long-term goals and reevaluate your use cases.

Successful GenAI transformation requires cultural adoption and organization-wide orchestration. Each position needs to know how to use it properly to be effective. Improving developer productivity requires equipping an entire team with the tools to allow everyone to move faster.

Without positional training, GenAI can have adverse productivity effects, as improper searches and unclear prompts can derail even the most experienced developer. In addition, our ways of working will need to shift to accommodate how this new technology affects the development lifecycle. According to Weldon, even “the way we work in Agile may need to be adjusted over time,” reevaluating sprint lengths to take advantage of developer productivity.

In conversation, Slalom senior director Miheer Munjal posed the question, “You don’t have to use GenAI to get to the solution, but if it helps you get there faster, why wouldn’t you?”

We know the competition is fierce, and in today’s world companies that move faster continue to gain market share over those that move slow.