We kicked off our recent Software Release Summit with a fireside chat between Honeycomb Field CTO and renowned developer advocate Liz Fong-Jones and LaunchDarkly CTO and Co-Founder John Kodumal. They talked broadly about industry trends and current challenges facing leaders, and shared practical advice for getting started with observability and feature management, and leveling up from high-performing to elite.
We've got highlights for you below, but as always, there's more to the conversation than we can capture here so we encourage you to check out the full chat. You can also catch up on all the Software Release Summit talks to hear more about the trends and technologies shaping the software development and delivery landscape, as well as get insights and best practices from leaders in the industry.
What challenges do you see engineering leaders facing in 2023?
"Macroeconomic challenges mean that software development leaders are being asked to do more with less, whether it’s having headcount freezes and not being able to hire as many engineers as you’d hoped for, or having to cut your cloud budgets," said Liz. Spending your way out of a problem and hoping that revenue will solve things later is no longer an option.
"Instead, we're having to be a lot more cost-conscious and are thinking carefully through things like, is this expenditure actually moving the reliability needle?" Liz explained. "Is it actually helping us deliver software faster and enabling us to cut back in areas where we're doing undifferentiated heavy lifting?"
Liz's recommendation is to focus on what truly differentiates your company and to bring in outside tooling that addresses common problems that are best solved by common solutions (heads up if you're playing Build vs. Buy bingo!).
Are you excited about any emerging trends or technologies around releasing software?
Honeycomb is an observability company, so it's not surprising that it's top of mind for Liz here. "I think that we are past the era of saying, 'Okay, we're just going to slap some continuous integration and continuous delivery on it.' … When you ship shit faster, you ship shit faster, and we need to impose quality guards," Liz explained.Â
"But, how do we actually measure and control what's happening in production?" Liz continued. "This is where we've really seen the story of observability come to life, when teams need to be able to understand and measure what's happening in production."
Once you're able to understand why your latest release didn't work, or why it was broken for a particular subset of users—observability—the next part of the equation is controllability. "Okay, you can see what's happening in production now," said Liz. "How do you affect it without having to restart the whole system? How do you make sure that you have the control knobs that you need to maintain system stability even in the face of an ever-increasing complexity of systems that we're building?" This is the power duo of observability and feature management that Liz is excited about.
Any thoughts about how generative AI might impact the software development and software release landscape?
"At the end of the day, an AI is not going to run your entire system for you. It's not going to respond to your pages," said Liz. What you can use AI for instead, is to automate things that are rote and boring for humans to do.
"I think the more creative uses of AI have the potential to create code that no one actually understands, that they just check in and hope that no one complains about it," Liz cautioned. "And when that does break, there will be no one—not even the person who originally checked it in— who actually understands what it's doing, why it's doing that, how it can be fixed."Â
Instead, Liz shared that her and Honeycomb's human-first approach to AI (as well as machine assistance in general) is to think of it as a tool to help, empower, and extend a developer's reach, rather than an omnipresent force outside of your control that replaces human intuition and context. "Think Mech suit, not Skynet."Â
How do teams become high performing or elite?
While observability and feature management are hallmarks of elite engineering organizations, Liz felt strongly that a faster release cadence is a prerequisite for realizing the benefits of those practices.Â
"At the end of the day, no amount of feature flagging or observability is necessarily going to help you if you are constrained in your ability to ship a longer standing fix to an issue by having a CD process that takes multiple days to complete and requires manual intervention… We really need to focus on those fundamentals of repeatable, automated software releases first, and then we can start layering in those additional degrees of visibility and controllability," said Liz.Â
Any advice for companies just getting started on their DevOps journey?
"With regard to observability, you cannot possibly go wrong with OpenTelemetry right now. There are a wide variety of vendors out there that all support OpenTelemetry. If you start using OpenTelemetry automatic instrumentation, it will start giving you out-of-the-box visibility pretty much regardless of what programming language you're using these days," Liz said. "And it will generate the data in a format that you can feed to any number of tools."Â
"With regard to feature management, you could choose to implement rudimentary feature management yourself and see how that goes to get an idea of whether that's right for you and your company," Liz said. But at a certain scale, once you advance to where you're deploying to thousands, if not millions of customers, she said, you will want the benefit of a feature management platform.Â
"What happens if your feature management service is unavailable? What are your fallbacks or how do you make sure that you are serving predictable results and recording what feature values were sent out? These are things that LaunchDarkly has thought about that you shouldn't have to think about," Liz said.Â
How would you justify the ROI of an observability or feature management tool?
"We are here to do the job of delivering software that delights our customers," said Liz. "And anything that gets in the way of that is something that we want to mitigate." If you've already invested in CI/CD, you then want to be able to understand what is happening with your users in production. "I don't want to be surprised by customer success calling me up and saying, 'Hey, there's this one user that is saying they're repeatedly encountering problems. Why is your dashboard green? Why does it say that there's no problems in the system?'"Â Â
If you're delivering software regularly, you also want control over reverting only the thing that's breaking, instead of dozens of people's commits. Beyond incident response, with robust feature management you are also empowered to experiment and take more risks. Liz said, "These are things that enable us to try things out on a smaller subset before we roll them out to 100 percent."
What advice would you give to engineering managers looking to empower their developers?
Liz suggested engineering managers seek to understand not only what their teams complain about, but also what pain points they have accepted as inevitable. What are the things that cause friction but seem like they could never change? "One classic thing that I harp on a lot is build times. It is a tax on every developer, the longer they have to wait to get feedback on their changes," said Liz. "That has really, really subtle and nefarious downstream effects."Â
According to Liz, it's critical that leaders get a sense of the friction that is standing between a developer writing something and it being used in the field, "and then investing in making that feedback loop shorter."
John and Liz also covered the tension between moving quickly and working with guardrails (particularly in regulated industries) as well as some great examples of dogfooding LaunchDarkly and Honeycomb together. You can catch up on those in the full fireside chat.
