CCP Games deploys on Fridays and does self-serve experimentation

CCP is a leading independent game developer that has been praised for its artistry, technology, and game design that facilitates emergent behavior, empowering players with compelling means of self-expression. Founded in 1997 on the principle of pushing the envelope and breaking new ground on all levels, CCP is on a mission to create virtual worlds more meaningful than real life. With the launch of EVE Online in May 2003, CCP established itself as a pioneer of cutting-edge massively multiplayer games, winning numerous awards and receiving critical acclaim worldwide. CCP is headquartered in Reykjavik, Iceland, and has additional studios in London and Shanghai. https://www.ccpgames.com/

More than a game, EVE Online is an elaborate virtual world that has persisted for nearly two decades, won Guinness World Records, and captivated millions of players across the globe. Since its launch in 2003, the game—a massively multiplayer online role-playing game (MMORPG)— ran on legacy systems. And CCP employed legacy processes to build, deploy, and operate their application.

Over the past five years, leaders at the company decided to modernize the game for the next generation of players. “We’re moving from metal in data centers to containers in Kubernetes,” explained Nick Herring, Technical Director of Infrastructure at CCP.

But their application modernization efforts went beyond just retiring servers. They were also determined to modernize their processes, particularly those tied to software development. They needed to shorten development cycles, reduce inter-team dependencies, and streamline software delivery. What’s more, they wanted to be more data-driven when building and shipping features. 

Feature management ended up helping transform the company’s approach in ways the team had not anticipated.

The platform could do much more than that. For instance, historically, deployments and releases were one and the same at CCP. But with LaunchDarkly's feature flags, they could treat deployments and releases as totally separate events. In Nick’s mind, such a decoupling would have a profound impact on how the team shipped software.

They could deliver and control features in production while running experiments on those features in one seamless workflow. This was a big draw.

At the time, CCP had a few homegrown feature toggle systems. But they weren’t designed to handle the sophisticated jobs required by the team. In the end, Nick chose to adopt a best-of-breed feature management and experimentation platform. 

In the past, developers often had to wait to ship their code changes until a big product release. Close to the first cut of a release candidate, several developers would merge long-lived feature branches with the mainline. This created points of no return in the software delivery pipeline. 

Moreover, given that each software update was rather substantial, and all changes were unveiled to all users at once, the team only felt safe releasing at specific times. They avoided shipping on Fridays, for example. Another reason for the team’s caution when deploying was that, if a bug arose in production, it was a pain to fix. In some cases, engineers had to re-resolve issues each day over the weekend. 

Now, with LaunchDarkly, they wrap code changes in feature flags, thus allowing them to deploy the changes to production servers without releasing to end-users. As a result, the team can aspire to trunk-based development. 

Each day, developers can conceal their changes in feature flags and merge them with the mainline. If the mainline happens to get deployed, developers needn’t worry as much about their changes breaking production; after all, the changes are hidden. Long-lived feature branches and painful merge events have faded away.

Another benefit of LaunchDarkly is it has enabled the team to disable broken features instantly. Now, when an incident occurs, rather than perform a laborious rollback or monkey patching, they can toggle a flag in LaunchDarkly. This capability has further boosted the team’s confidence—and speed—when deploying.

Developer velocity, productivity, and satisfaction have risen. Engineers have also gained more autonomy. With feature flags acting as safeguards, developers are free to deploy on-demand. They’ve also been granted more autonomy as a result of the auditability of LaunchDarkly. CCP leverages LaunchDarkly’s integration with Slack to send alerts to a dedicated Slack channel every time a feature change is made to EVE Online. This paves a clear audit trail.

Another reason developers can ship their changes more frequently is they don’t have to wait on each other. Once an engineer has completed a feature, they can deploy it immediately. Later, someone from the publishing (release) team can go into LaunchDarkly and turn the feature on for the right users.

All in all, CCP has become much more agile when it comes to software delivery.

Having implanted feature flags in their software delivery pipeline, CCP then added experiments to the mix. Here is a typical workflow the team follows in which you can see the harmony between flags and experiments.

A developer builds a feature and wraps it in a flag. They test the feature on live production servers but without revealing it to users. After observing how the feature interacts with the rest of the application, they then layer on an experiment to see which feature variations drive the most engagement. Note: someone closer to the business units, like a product manager, can also run an experiment in this manner. The team may conduct the experiment with a small percentage of users. Upon gathering results, they might use LaunchDarkly Data Export to export the data to their analytics stack for further analysis. Once they uncover an actionable insight, they ship the winning feature variation with a few clicks. This all happens in LaunchDarkly.

In one example, the team used LaunchDarkly to drive real business results. EVE Online has a recommendation engine that suggests actions for new players to take. The team decided to use granular targeting rules in LaunchDarkly to tailor recommendations based on different player attributes. They then weaved in experiments to see how various recommendations affected user behavior. After fine-tuning the feature based on the data they gathered, they delivered a new and improved recommendation engine. This led to a notable rise in median session length in the game. It also created a more personalized experience for each player. What’s more, it resulted in the creation of the AIR Career Program, an entirely new feature that helps increase user engagement.

LaunchDarkly has helped enable modern software development practices and product experimentation at CCP. But it also helped the team modernize their application infrastructure.

As a part of their digital transformation effort, CCP wanted to replace legacy, on-premises components of their stack with cloud alternatives. Such a major transition presented a lot of opportunities for things to go wrong. But the team was able to leverage feature flags to mitigate risk during the process. They reduced workloads into smaller chunks and migrated them gradually from legacy systems to the new cloud-based ones. 

Since adopting LaunchDarkly, CCP has made great strides in modernizing their development processes. Moreover, they’ve enabled a wide range of people to run experiments that yield tangible business results. Finally, Nick and his team have been able to safely move application workloads to the cloud.   

CCP was delivering a masterful game experience long before LaunchDarkly arrived. We simply helped take them to the next level.