How HP standardizes and scales releases with LaunchDarkly

HP, Inc. is a technology company that believes one thoughtful idea has the power to change the world. Its product and service portfolio of personal systems, printers, and 3D printing solutions helps bring these ideas to life.

Moving to a more frequent, standardized software release schedule at a Fortune 500 company takes a large, coordinated effort combined with a lot of willpower.  

When Alan Santos, Research and Development Manager at HP, arrived at the company in late 2019, he wanted to speed up releases. 

Santos works in the company's Print Software division, and oversees many of the consumer and business portals that customers interact with the most. One of Santos' early observations was that the company had complex software running in multiple environments: development, integration, pre-production, and production. Managing releases and their dependencies across those stacks was challenging. 

Luckily for Santos, HP was already way ahead of the game. The company had been proactive in moving towards addressing those issues, undergoing an internal effort to unite disparate teams and standardize processes. 

As part of this project, HP was moving to a single ecosystem to connect all its clients, printers, and computers to a common cloud infrastructure. Looking at the considerable number of projects being worked on by multiple teams distributed across the globe, the need for this transition had become self-evident. 

For years, the segmented approach to development had been working at HP. Individual development teams—located everywhere from Shanghai to Korea to India to Brazil—were able to select the tools and processes that best suited their needs, and go forth and execute. Even though the efforts were relatively siloed, each team's scope was manageable, so the risk was low.

However, the future was a more cohesive, and a standardized, platform-based approach, where everything flows through a common cloud infrastructure, including deployment patterns. Reading the tea leaves, HP put its considerable resources towards achieving this goal.

"As that came into reality, it created an environment where all of these teams that used to play individually now had to work together," says Michael St. Laurent, Chief Architect of HP's Print Software division. St. Laurent's role provides oversight across all of the software organizations—that includes everything from cloud services to client applications to print drivers—and ensures the company is building a broad ecosystem where many systems are connected via the cloud. 

On the ecosystem approach, St. Laurent worked closely with Brent Foley, a DevOps engineer at HP, who is focused on the company's infrastructure. Initially, getting all teams working on coordinated deployments took a lot of manual effort, meetings, and alignment.

When it came to deployments in HP's Print Software division, one of the areas in need of updating was process standardization, including the tools being used. 

As an example, for years prior, development teams at HP had access to LaunchDarkly. Foley remembers initially helping select LaunchDarkly for HP because it was one of the biggest players in the feature flagging space, and the company was interested in the concept of progressive delivery. "It all came together because we were big proponents of the whole idea of shifting left and progressive delivery," Foley says. "LaunchDarkly had a big voice in that space."

Given that different engineering teams preferred various methods at HP, sometimes LaunchDarkly would be used on projects for feature flagging, but other times, teams would use a mix of config files or home-cooked solutions. 

In order to get teams more aligned when it came to development and deployments, the Print Software team had to get more prescriptive with the tools being used. The technical leadership team made the decision to mandate LaunchDarkly for featuring flagging within the Print Software division. That wasn't enough though; HP wanted to empower the teams to be successful. So the team went about creating a 14-page guide of best practices for using LaunchDarkly at HP. 

The combination of selecting LaunchDarkly as the tool of choice for teams for feature flagging, and the creation of a documented guide for using the platform, proved powerful for bringing teams up to speed.

"In terms of adoption," St. Laurent says, "I think recognizing the needs of the organization, along with creating of the collateral materials—which were the standards—gave us something that we're able to go back to the team and say, 'You need to use LaunchDarkly if you're going to do this, and by the way, this is how you do it.' It was those two things that accelerated it."

Prior to standardizing with tools such as LaunchDarkly, there were often concerns within Print Software at HP that something would go wrong with a deployment. In the past, if something unexpected happened during a deployment, it could result in multiple hours to prepare for another deployment as the team reverted and monitored the system for any red flags.

Now, with LaunchDarkly being part of HP's standardization efforts, when incidents happen during a deployment, the team can undo changes immediately. The developers who are working through things can typically release, verify, and then go off and get some rest or move on to the next thing. This has allowed the team to focus more on delivering than recovering from incidents, and it's made them more confident overall.

As Foley puts it, the company is looking at ways to track the number of rollbacks teams are making. "If you're doing all the best practices like feature flags and some other things, we shouldn't have a situation where you need to roll back a version of the service," he says. "If we see that you're doing a rollback once a week in production, you're doing something wrong."

For Santos, he no longer feels as dependent on other teams for software delivery. He can deliver front-end features that are, in some cases, months ahead of the backend systems, and vice versa. And then when those dependencies are ready to release to production, there's no need to run through the whole deployment process again. The flag with a specific feature that's already been validated and tested can just be toggled on.

Along with the array of internal efforts put into standardizing processes, LaunchDarkly has also helped simplify some of the initial complexity Santos found with software running on multiple environments. Now, he can simply push a button to enable and disable flags by stack and at runtime.

"The program managers can plan dates to enable feature flags using LaunchDarkly in each stack," Santos says. "On the quality standpoint, the QA team is able to validate a feature behavior in runtime changing values. Also, the team can switch back to a feature behavior without doing code changes."

And since LaunchDarkly has helped teams decouple releases from deployments, their release schedule has quickened. A little over a year ago, the team's releases were all full deployments. Now the team has split its release cycles into deployment phases and release phases. As an example, St. Laurent said some launches have been shaved down from hours to minutes.

Looking ahead, Foley is hoping to get some of the teams benchmarked around DORA metrics, and eventually graduating some to the level of elite performers. "We want to get teams using all the best practices and tools, but LaunchDarkly is a big part of that vision and that roadmap to get there."

Santos says he estimates LaunchDarkly has sped up his development and QA process up to 15%, at least. He thinks back to one of the initial demos he conducted of LaunchDarkly, when teammates weren't very familiar with the solution, showing how features can appear and disappear instantly.

"Everyone was going crazy," he says. "It's something amazing."