Fast vs safe: The great dilemma in modern software development

Today, software developers are stranded between fast-and-risky and slow-and-safe. Here’s why.

1. Fear of production creates a logjam of pre-production testing

Pressured to deliver perfect, bug-free releases, developers endlessly vet code in pre-production staging environments, then let it sit with quality assurance (QA). This artificial testing eats up considerable resources and slows developers down.

Testing in production can accelerate the process. But developers tend to avoid testing in production due to the perceived risks of the practice. Ironically, any time a developer deploys code, they are, in fact, testing in production. They’re just doing it with their entire userbase. And to be fair, developers typically lack the control required to test in production deliberately and safely. Nevertheless, development teams will continue to slog if they excessively test code in staging.    

Every developer tests in production... Some admit it and plan for it.

3 simple principles of modern software development:

• You can’t release a feature to production until you’re sure it works.

• You can’t be sure a feature works until you’ve released it to production.

• Go to 1.

2. Errors bring down the application, disrupt customers, and take ages to fix

Most developers lack the ability to quickly revert to a known safe state after a bug or outage. This forces teams to maniacally test code before a release to reduce the risk of impacting customers. 

When a bug causes an outage, software engineering teams often need to perform a manual rollback to stop the bleeding. That means routing all production traffic—potentially millions of users—to an older version of the application and muscling changes through the deployment pipeline. 

It’s a cumbersome, stressful, painful exercise that developers will go to just about any lengths to avoid. Even worse, revenue losses and user frustration will mount as developers scramble to bring the application back online. Users have little patience for downtime or broken features.

Average cost of application downtime, depending on the size of your business: $137 - $9,000 per minute

3. Homegrown feature flag systems devour precious time

Many of the challenges we’ve outlined can be mitigated with feature flags. Feature flagging is a modern software development concept that allows developers to enable or disable a feature without modifying the source code or redeploying their application.  

Software engineering teams adopt feature flag systems to increase development speed and reduce risk. But, with homegrown solutions, the additional time spent building, managing, optimizing, and repairing the tool can easily cancel out any gains in developer efficiency.

Typically, homegrown solutions lack an intuitive user interface and require developers to create duplicate flags for each application environment, platform, and service—or risk providing a poor, inconsistent user experience.  Not to mention, developers must keep their homegrown tool from causing data breaches, incurring excess technical debt, and degrading system performance. 

It all adds up to a significant burden on developers—one that pulls their focus away from creating features that serve customers and generate revenue.