Galaxy Talk: Supercharge Your Release Pipeline With These LaunchDarkly Configs

As organizations grow, it’s tempting to stick with the way things have always been done. 

And yet, stepping back and asking questions can yield big results.  

In this session from our recent user conference, Galaxy, Caley Brock, a senior software engineer at VTS, talks about some of the improvements that changed her team's process and enabled faster shipping and more streamlined testing.

Transcript:

Hi, everyone. My name is Caley Brock, and I'm a senior software engineer at VTS Canada. Today, I'm gonna share how VTS supercharged our release pipeline with the help of these LaunchDarkly configs. Today, I will cover an intro to VTS in the process that we call AsyncQA. We'll then discuss the LaunchDarkly tools that we use to help us with our process, specifically tagging, integrations, and permissions.

VTS is a modern commercial real estate platform. We've been around for almost eight years and have an engineering team of over 80. I work on VTS' main monolith application, which powers our core product and contains plenty of code, both old and new.

Over the past year at VTS, I've personally become really interested in how we release software. I've had a few release experiences that sound familiar to probably a lot of us. They utilize multiple dependent flags which despite my best efforts, of course, were challenging to keep track of and hard to clean up. We ended up having to maintain extra complexity in code and use spreadsheets and other documents to just track the status. I know that with the right tool and the right process, we can do so much better.

Luckily for us, one of our newer co-bases at VTS was already using LaunchDarkly for their feature flag management and they really liked it. After some research, it looked like LaunchDarkly could be a good fit for us too.

So, this past Fall, I created a proposal to expand our use of LaunchDarkly into our main monolith application as our feature flag management tool, replacing and switching away from our existing tool that we used in the past. At the same time that I was working on this proposal for LaunchDarkly, our organization was in the middle of trialing a new way for how our manual QA process fits into our release process.

So, about six months ago, this is what, uh, my team would do to release a feature. We'd write our code, we'd write our test, and then we would create a review of, uh, with a cust- which is basically just a custom environment, uh, for just that branch of code that holds a feature that we're working on. Then, QA would go in that environment, test everything, and only after the approve would we merge and deploy that code. So, we'll call this process our traditional QA process. 

The process that we were introducing and that we use now, we call asynchronous QA or "AsyncQA" for short. So, this time, what we do is our code and test get written behind the feature flag and then immediately get merged and deployed. So because these changes are safely hidden away, QA can test anytime and on an integrated environment or in production.

There's no more need for this one-off environment that we have in our traditional QA process. And once they take their time to approve and test everything, they then go and turn that flag on for all users on productions. And then, the ticket comes back to the developer and we would go and remove a flag from code and from our tool.

So, we start trying out this new process team by team. Um, my team tried it out and we noticed some really significant wins. 

So first of all, we no longer have to wait for QA to merge which meant no more stale code, um, and then allowed our code to be integrated much faster. We were also tracking our cycle time, which is just the time from starting a ticket to when the users see value. And we noticed over 20% reduction in our cycle time while still maintaining quality metrics. This meant that users were seeing our changes faster and we're not, we're still seeing no more bugs than before, which is great news for us.

We also said that QA could be more flexible. They were testing on integrated environments things together, which made sense. It was better for testing, um, and we didn't have to maintain these environments. But of course, as with any new process, it also introduced its own subtle complexities. 

The most obvious is that we started seeing way, way, way more flags, pretty much a flag per ticket for tickets going through AsyncQA. And because of this, it was easy to get disorganized fast. It was not always clear which flag did what and we were worried about the clean up of flags slipping through the cracks.

QA also now manages these releases which is awesome, they have the context to be able to do this and it made a ton of sense. But, it meant that we have more hands touching production states and even more production states to manage.

It turns out that our feature flagging tool and our AsyncQA process were really reliant on each other. In order to gain the benefits of AsyncQA—like better cycle times, better testing—our feature flagging tool had to be able to handle this kind of growth in complexity. So, we took a look at how our new tool, LaunchDarkly could help mitigate some of the new challenges we're seeing with this influx of flags from AsyncQA.

The first thing we set up was a system for tagging. So, you can add any custom tag to your flags in LaunchDarkly and we definitely take full advantage of this. Right now, we have two types of, uh, tagging. So, one is ownership tags, so these denote what team is responsible for that flag and it typically looks like something like team dash team name. And then, the other is process tags, so denoting, um, which kind of release process this flag or this ticket or feature is going through. So, in this case, what we're talking out now, AsyncQA.

And once you start using, uh, tags consistently and have something set up, it makes it really easy to go into that LaunchDarkly feature flag list and be able to use the filters. So for example, our QA team members can easily go and see what flags their team is releasing through AsyncQA, what's been launched, what's still up to test, and it's right there and easy to find.

The second thing that we set up, uh, that was really helpful were integrations. So, I think that integrating new tools into existing tools is a huge step. It really helps new tools feel normal, uh, because you don't have to go looking for it and it already feels like it's part of the process. Um, and it builds a lot of confidence. 

So, during integration was especially helpful for AsyncQA because you can create flags and see the flag state from within the ticket. So, there's no mixing up what the flag does, which one does what, and we don't have to search for the information we need. Whether you are in LaunchDarkly or you are in JIRA, you know exactly where things tie together.

We also, uh, made sure to install the LaunchDarkly Slack App, um, so we started by having a log channel for, uh, our LaunchDarkly production environment for the project. Um, and that kind of has a log of messages of like anytime something in the state changes in correction. This was really useful especially when people were learning the tool. It built confidence because it was an extra check and extra layer that confirms that users were doing what they were expected to do. But my personal favorite feature in Slack is that it'll send you messages if flags have been launched, essentially, uh, if a flag has been serving the same state to all users for at least seven days. Um, and this is just an indication that we can probably remove that flag from code in LaunchDarkly.

It's an automated reminder to get rid of our flags and it doesn't- it makes sure that we're not letting those things slip through the cracks. Um, this is so great for us because now we don't have to in line remind each other; we have automated tooling to help us with that.

We are currently in the process of this last upgrade, uh, so we wanna restrict our right permi- permissions on production to only the groups that need it. So right now, LaunchDarkly, everyone has read and write access to all environments and all flags, um, but what we realized, now that we're kind of solidifying our process, is that only some people actually need that production access. And we wanna make sure that those are the people who have permission to do so and change things. So, for flags tagged as AsyncQA, that's our QA team. This is a small change, um, but it's another step towards reducing, um, error that, any human error that can come in and ensuring quality, um, as we start to expand and use even, this tool even more.

So, LaunchDarkly is now setup to support our AsyncQA release process, uh, which is great. There's no more mistaking what flag belongs to what ticket. Flags are easy to find and manage, they're safely permissioned, and we won't be able to forget to move- remove them when we're done. It's super clear what their ownership is and who needs to do what. With these changes, we are even more confident that we can switch to this process and take advantage of the benefits of AsyncQA.

Leveraging organizational tools like LaunchDarkly is key to successful organizational changes. At the surface, feature flagging may seem like a pretty technically straightforward thing, you know, series on off, on and off toggles. But without the organizational tools that come with LaunchDarkly, making this process change to AsyncQA would've been a much bigger lift to get us to the confidence level that we are now. We're still in the early stages of using this tool and this process but we're excited to see how they can continue to expand and support our teams.

Thank you so much for listening and major thank you to my teammates. This project was and continues to be a collaboration between engineering and QA and I'm so, so grateful to get to work with all of you.

And if your team is a little ahead of VTS in terms of your LaunchDarkly adoption, I'm curious what your learnings have been especially with your setup. Reach out, let me know, and I will see you all in the Q and A.

From LaunchDarkly tips and tricks to thoughts on migrating from homegrown systems, check out the rest of the talks from Galaxy.