Galaxy Talk: Fitness Functions and Flagging Conventions

Welcome to our ongoing series spotlighting different talks from our interactive user conference, Galaxy, which was held in April. 

“From chaos comes order” is a famous saying. That may be true for some things, but when it comes to naming conventions, feature flag management, and tech debt, chaos can create more chaos. 

In this talk from our recent user conference, Galaxy, Mark Burry, senior developer at iPipeline, recounts the journey his company embarked on with feature flags, and how it eventually found fitness functions and flagging conventions as best practices. 

Transcript: 

Hello, there. Thank you for joining me to discuss fitness functions and flagging conventions. My name is Mark Burry and I am a senior developer at iPipeline.

Honestly, even though my Twitter handle might make you think otherwise, I do not work for LaunchDarkly. That is just called @launchmarkly.

Anyway, uh, I have a question for you. Have you experienced any issues with your feature flag naming convention, flag management, or technical debt clean up at any point? We certainly had, so we implemented a range of fitness functions and processes to ensure that our flags were well maintained and loved throughout their life cycle.

Okay, Mark Burry, so who's iPipeline? iPipeline is a strategic partner and technology provider of SaaS solutions. We are passionate about what we do and take pride in leading the information technology and services industry in providing web-based software tools that drive straight through processing for the insurance and financial services markets on a global basis.

Here in the UK, our products cover more than 35 leading product providers across protection, annuities, pensions, bonds, and equity release services.

So our LaunchDarkly journey. Well, it started back in 2016. We have been using LaunchDarkly for five years now, and it is one of the pivotal pieces of software that enabled us to move to a continuous delivery model.

In that time, we have created 200 plus flags across our US and UK projects. These projects have multiple development and production environments managing different targeting and release cycles.

We have slowly but surely developed a test in production mentality over the years, due again, in part, to using LaunchDarkly and the safety net it provides in production environments.

And have 330,000 plus daily flag evaluations across all the features in our US and UK products.

So fitness functions and flagging conventions are the solution, but to what problem? Well, um, after you've been blown away by the power of feature flagging, having deployed your first flag into production, implementing feature flags soon becomes second nature. Then at some point, you will look at your dashboard, realize that you've last sight of what each flag is doing, how long they've been in production, who is managing them, when they're gonna be removed, et cetera, et cetera.

This happened to us back in 2019 and we had an "oh no moment." It became very clear we had a flag management problem. We had 300 plus flags in production with some of them created, deployed, and released up to 18 months earlier. We had started our LaunchDarkly journey under the pretense that our flags would be temporary and removed shortly after a project was completed, yet here we were in 2019 with flags 18 months old and the projects that had created them finished long ago.

For us, this was not a good place to be in and soon we found ourselves in a situation with nested flags, inconsistent naming conventions, misunderstood targeting, and a complicated tech deck cleaner ahead of us.

So what did we do next? We implemented fitness functions and flagging conventions, so we identified if we didn't act- if we did action it, it was only gonna get worse and worse. Um, we had a high number of users with the ability to create flags and soon it became cowboy land concerning naming, tagging, et cetera. The issue is certainly human. It isn't with LaunchDarkly, but rather how we implemented our flags.

So what is a fitness function? So, an architectural fitness function provides an objective integrity assessment of some architectural characteristics, easy for me to say. Essentially, a fitness function is something that puts checks in place to assess how fit a purpose a piece of functionality is, and ideally these checks are automated.

This is exactly what we did. Implementing fitness functions that assess each our flags throughout their life cycle from as early as the naming of the flag all the way through to the deletion of the flag.

We had several key areas, um, that we focused on. These included code references, deploying a tagging, uh, developing a tagging strategy, running automated assessments on our flags, and implementing we-web hooks to provide continued visibility.

The process has evolved over the years where these key areas have allowed us to maintain focus on our flags throughout their life cycle, even when projects have moved on. No flag is left behind.

So, code references really do rule the world for us, uh, and they are great and a real game-changer. The key to our success with code references and flag removal was a fundamental mentality change. We took a stance to always be thinking about flag removals.

During our "oh no moment," we had flags that were implemented by staff members who were no longer with the company, which caused issues with understanding the logic behind the implementation. So when bringing a flag into code, uh, we look at how to make it as easy as possible for someone to remove the flag.

We started block commenting our changes so that it could be easily removed by anyone regardless of their knowledge of the flag. The example on screen is very straightforward logic, yet we are to quickly identify what code is related, uh, to the feature flag and what should be removed at a later state. This block commenting approach works for simple and complicated logic across multiple files within one repo or across several repos, and it played a huge part for us in mitigating the risks around flag removals later on.

So the power of tags. The power of tags really can't be underestimated. When combined with our fitness functions, they help maintain our flagging conventions. We have a number of standard tags that we add during flag creation and throughout its life cycle.

The project/team tag is used to understand, well, you guessed it, which project or team is responsible for the flag.

The repo tag, again, responsible for identifying which repository the flag is in.

This becomes key when used with our fitness function to monitor if a flag is referenced in the master branch.

Then we have the last two, which are removal in progress and awaiting third party. We want our flags to have a short life cycle of around 90 days. The removal in progress and awaiting third party give us a tag to validate against to ensure we are keeping within that 90 day limit.

All of these tags are used to group flags together within LaunchDarkly and quickly review what flags are related to a project, repo, whether something is awaiting on a third party, or if a flag is in, um, removal.

So the fitness functions themselves. So code references and tagging policies are vital for our fitness functions, but they are merely the ingredients to our fitness function. So the flagging convention we, uh, developed back in 2019, and these fitness functions help ensure that we continue, to continue to adhere to those processes.

We have a custom-built application which runs on a daily basis evaluating our flags. This application utilizes only a small number of the API approaches available through LaunchDarkly. We retrieve all the repositories that code reference is aware of and evaluate all the master branches. This is where the repo tagging is crucial as it is used to evaluate if a flag is in master.

We retrieve all the projects held within LaunchDarkly or flags contained within each project along with its status. This is then used to run our validations against to ensure that our flagging conventions have been adhered to.

So once all the data has been retrieved, we then run our assessment of architectural characteristics, back to that tongue-twister of a quote. These characteristics range from simple absence of a description or tag to logical checks against created dates and flag life cycles. We are able to review when a flag was last evaluated in a live environment, and if it has been moved, uh, marked for removal, we can then raise an alert to archive a flag at the appropriate time, should it not have been evaluated for two weeks, and is no longer referenced in the master branch. These assessments trigger a notification should any one of them fail and this is where the final piece of the puzzle comes into play.

So webhooks and Microsoft Teams, other platforms are available. Um, the fitness function just discussed, uh, they trigger a notification, and for us, it pushes into Microsoft Teams channels. Um, this notification will continue on a daily basis or weekly until the issue is resolved. Issues with naming conventions, uh, tagging, uh, are daily reminders that require attention. However, a flag that is awaiting a third party is out of our control, so we'll only get notified once a week. These notifications ensure visibility of any flag issues is not lost and will continue to be discussed, uh, until the issue is resolved.

Last year, the LaunchDarkly integration with Microsoft Teams allowed us to create an audit channel which raised visibility, uh, further concerning any changes to our feature flag. This has proven incredibly valuable with the rollout of the request approval process within LaunchDarkly. We implemented that fitness function into our process straight away to ensure changes in production environments were correctly managed.

Those are the four key areas we developed back in 2019 and have so far allowed us to avoid a second oh no moment, maybe just jinxed it. However, in summary, what do I hope you take away from the presentation?

Well, implementing code references, um, and always be thinking about, uh, flag removals. Code references are amazing, definitely worth your time. They allow you to track where a flag is used and help with identifying when it can be deleted. Thinking about flag removal as early as possible with block commenting allows you to avoid any issues later on.

Simply put, tagging your flags is crucial to any fitness function checks as well. So create a tag policy, um, is key for grouping flags together even in LaunchDarkly outside of any custom fitness functions. However, for us, they were key in developing, uh, checked that we can do within our fitness functions.

Um, and then the fitness functions themselves, they allow you to adhere to a flagging convention, so create a naming convention, life cycle, time frame, et cetera, and then wrap an automated fitness function around it. Um, this way you can ensure that you adhere to those, uh, naming conventions, uh, throughout all of the life cycles of the flags.

And then finally, enable constant visibility of your flags through webhooks. Never leave a flag behind. Utilizing fitness functions to evaluate your flags and webhooks to notify interested parties of any issues is key to managing your feature flags.

So I hope that I've given you some food for thought regarding your flagging conventions and implementing fitness functions. Um, haven't gone into too much depth in the slides, but if you have any questions, then please, feel free to reach out.

Uh, thank you for your time and enjoy the rest of the conference.

From launching LaunchDarkly at large organizations to supercharging your release pipeline, check out the rest of the talks from Galaxy.