3 keys to maximize the business value of every product feature

Stefan Thomke
Professor, Harvard Business School

1. Gut decisions + poor metrics = poor results

Organizations often get stuck in the “just keep shipping” mindset of continuous delivery, leaving them little time or resources to determine what customers want or how best to give it to them. Without data to inform decisions, software teams are forced to go with their instincts. Unfortunately, when software development companies “trust their gut” about customer preferences, they can be wrong as often as 90% of the time.

Plus, even when data is available, it frequently isn’t useful for driving business impact. Too often, teams settle for engineering metrics like lead time for changes and deployment frequency. While these measurements are helpful for meeting efficiency benchmarks, they have little-to-no effect on the customer experience. Decisions about product and feature changes should instead be informed by customer behavior analytics and collected feedback. But this data isn’t always available to developers and engineers.

2. A bad back-end spoils a good front-end

To offer the best customer experience, your application performance must be top-notch. This requires the ability to run A/B tests and other experiments on back-end infrastructure, components, and services. While solutions for cosmetic front-end experiments abound, few exist for the back-end. 

As a result, developers, platform engineers, and SREs lack insight into how new software changes impact application reliability. What's more, they lack the data they need to fine-tune features with a view toward optimizing system performance. Frankly, it doesn't matter how good your front-end features are if the underlying back-end components are glitchy and unreliable. Until product delivery teams can run effective server-side experiments, their product experience will likely fall short of its potential. 

3. Silos and unscalable processes hinder experimentation

At many organizations, the tools and processes for running a product experiment are disconnected from the tools and processes required to deploy the winning treatment of an experiment. That is, running experiments and deploying experiment results require separate implementations. This disconnect leads to fewer experiments, less engagement data, and a longer time-to-value.

Teams will often run small, discrete experiments without much trouble. But as they attempt to scale experiments, the work becomes more difficult, the process grows convoluted, and the communication pipeline starts to break down.

Lastly, experimentation tooling that fails to integrate with your tech stack causes yet more problems. You’re forced to stitch data together across silos, deriving unreliable insights at a pace that’s far too slow to meet customer demands—and with a process that’s far too complex for non-technical users to leverage.

All of the above challenges put continuous experimentation and optimization out of reach.