Backyard Chat: Progressive Delivery

(upbeat music) - Hello and welcome. I'm Adam Zimman, VP of. Platform here at LaunchDarkly. I'm joining you from my wonderful backyard. And... So with that, comes all of the wonderful noises of nature and remodeling in San Francisco. And I'm also joined here by James Governor, the principal and founder of RedMonk, joined from his backyard, in wonderful sunny London. So James, thank you for joining me.

Good to see you. And audio is important but frankly so is getting outside in these rona times, we're spending all our time inside so blame me if the audio is giving a little bit of background noise. But I just think, let's get outside especially because, as we are here in a bit. The first time I really got to properly hang out with Adam, it was indeed in your wonderful backyard. We were having a little bit of Vermouth. And I just thought that given this story, this journey we've been on, why not? Why not record something in the backyard and get outside in the sunshine and enjoy each other's company?

Absolutely. Why don't you kind of, fill folks in, obviously, I've heard this but I'd love for you to tell your story--

So basically, like many good stories, it begins with rage. So I was at a Kubecon in Copenhagen, in 2017. And there was a lot of talk of service mesh. And it was like... A lot of cheerleading Istio, Istio, Istio. And like, nobody could tell me what it was for. Didn't matter who I asked. It just seemed to be a very circular conversation. And occasionally they'd say it was about observability. They might say it was about end-to-end encryption. They might say, it's a sidecar. So there was a sort of why are we talking about this stuff? And it was just... Honestly, it was irritating me. So we have rage coding, obviously, I guess in my case as an industry analyst, what I did was rage analyzing. And so a few months later, I was in San Francisco, and I was meeting Microsoft, and there's a great guy there, Sam Guckenheimer. And I actually just wanna say cheers Sam, you just retired. And, yeah, here is to you sir because it was through talking to Sam about the way that Microsoft worked, that I got really excited actually about... It suddenly dawned on me that there was a reason to talk about this sophisticated sort of networking that you could do with a service mesh, that you could do with modern architectures, that was really relevant to kind of user needs. And what he told me was that Microsoft does everything from the perspective of progressive experimentation. So they would always work out a cohort. It's like who's gonna be affected by this service? We can't roll it out to every customer worldwide. We wanna choose, maybe it's gonna be internal users first, maybe it's gonna be a particular geography. But its the... The notion of really understanding the blast radius of a service, before you roll it out into general production. And that got super... Got me super excited. I kind of came out of the room. I was buzzing, I was like... 'Cause it dawned on me it's like wait a second, progressive delivery.

We've had CICD, we've got companies that are kind of living in the future. They're doing continuous deployment. They're all over this. But one of the things that we haven't really been talking about at the enterprise level, and in terms of some of the, as we mainstream that idea, is this idea that indeed, we can do things like canarying. That we can think about the cohort, and then for obvious reasons, when a few minutes later, and it literally was that. I came out of the meeting, I was buzzing I was like "This feels like it could be a good idea." And I went to hang out with Adam. And he was like, "Ah, yes, progressive delivery." And so I was... We began to say progressive delivery, progressive delivery. Because clearly feature flags are one of the underpinning sort of technology approaches in feature management, about thinking how can we really understand the impact on overservice on users, before we roll that service out?

The thing that I found most compelling when you started kind of talking about this was the overlap in a lot of the conversations that I've been having with our customers, with others in the industry, with Sam as well, 'cause, we've been working with Microsoft for a long time. And, realizing that there were... These kinds of key aspects that we were talking about, that were really absent from the continuous delivery model, and by no fault of the individuals that created that model. It was more a function of those individuals were really looking for how do we optimize for modern application development in kind of a greenfield opportunity? And I think of this is, kind of new web-scale, applications or services that are being built from the ground up with modern tooling, whether that's, going as far as service mesh, or even just from a perspective of people that grew up in a world where the only infrastructure they ever had was on a cloud service, like AWS or GCP, right? If they were... or Azure. But, ultimately, it's one of the things where, their concerns, their thought process around delivery was so different than what I had experienced in running Global. Services at VMware, right? Where we did have concerns about physical servers. We did have concerns about physical network configurations that we were managing, or the ability for... like eventual consistency, means something very different when you were the one who actually owns the timeline to reach it, right? And I think that, more importantly than that, it was the realization that a lot of these ideas around continuous delivery, were really in support of organizations that were delivering services. That they had reasonably good confidence that the consumer audience, the customers, were willing and able to take change at a rate that was dictated by the company as opposed to a rate that was dictated by the business. And so this was something that was in contrast to what I'd experienced working with customers in financial services, or government sectors in particular, where they had strict regulation around the timeline, the cadence, at which they were able to accept change. And so when you and I started talking about this, it was just like... There was such an immediate kind of melding of what I was seeing, and what you were hearing and what you were saying. It just made sense for us to kind of start sharing this with some more, kind of formalization, some more terminology, right? And that was I think, how we landed on these kind of two core tenants of progressive delivery, being this notion of release progression, which is, directly in line with, what you're talking about. And this idea that you can actually have a controlled cadence, right? So it's not the continuous delivery aspect of (fingers snapping) everything as soon as it's deployed, is being shipped. And then the second aspect, which was this notion of delegation, right? And this was the one that I think, I brought up to you as being, something that I was seeing and I think you agreed with. It was the ability to actually control who gets to actually turn those features on and off, and for whom?

Yeah, that was what excited me. I mean, basically both of those points. One is abundance. Back in the olden days, and it's not so long ago, we didn't have abundance. We had a permission-based IT economy and it took a while to get things provisioned. You had to wait for your server, you had to wait for your database, you had to talk to the DPA, before you sort of started making changes. And obviously, we've moved quite far away from that because of this abundance of resources that you talk about in terms of what we've had in the cloud. And that sort of led to a lot of these thoughts and approaches around cloud native. But what you really nailed for me, was this fear in the customer. When you talk to them, as soon as they hear CD, they're like, wait, what? Does that mean I'm not in charge of when we roll out the service? And that's legitimately a scary thing. So when I got excited, and I've been talking to you as I progressive delivery, then I found I was going in to talk to. As you say, people in regulated industries, people with a really strong consumer focus, where the business has to be in charge, when the service is rolled out, rather than it being something that we can just rely on IT to do. That's not alignment. It's really about that product management focus. Business decides when the service is activated. The developer decides when they deploy and I think that the point you made about decoupling those two things, super exciting.

When we were first talking about it, it was, I definitely was like on this kind of mantra of the separation of deployment and release, right? And kind of being able to articulate this idea that you deploy when you want, and you release when you're ready, right? And this was the... I think was something that was very similar to what I'd experienced and seen at GitHub, where they had some fundamental use of feature management or feature flagging to be able to make sure that they could merge back into their trunk or mainline branch. But that didn't mean that it was visible, right? And so they had this notion of like a hidden switch or on, off, but it was actually very binary. And it was also something that it was very much controlled by engineering or extraordinarily technical product management. Whereas, what we were seeing more and more from the large enterprise customers that we were working with, was this notion that they needed greater granularity than that, right? It was this notion of percentage rollouts, like Microsoft was doing. It was this notion of the delegation to a business owner, as opposed to just another technical resource and wanting to make sure that we could account for those. And this changed my language subtly where it's deploy when you want, which is still kind of key to innovation, the key to engineering happiness if you will, of being able to move fast, right? But the release side changed to release when your customer is ready. Because immediately when you add that notion or that context of the customer end to your point, it's like, suddenly that's a business choice, right? That's not a technical limitation or a technical, forcing function. It is a business decision that you are actually making actively.

One of the good examples actually and I think this really ties into the kinds of things you're talking about, is looking in the way. Stripe uses feature flags. So we think about feature flags very often as something goes wrong, let's stop something happening, right? For Stripe, there's a slightly different focus. So they've got a thing where, okay, if for them if a circuit breaker kicks in, that can be hugely problematic. For them, it's about turning off the circuit breaker to ensure that the transactions continue to happen. And that's a sort of, again, a really subtle shift, or a dramatic shift in some respects. But it is all about making sure that the business is in control. Because, if we're talking about financial transactions, they have to go through.

What I've realized is this is what's kept a lot of those businesses, from being able to move from kind of a strict agile methodology to a world of continuous delivery, right? Because they've viewed that as an introduction of risk, and a loss of control.

Absolutely. It's all about control. And I mean, as I say, from my perspective, it was when I started talking to enterprises, when they have progressive delivery, they're immediately like, "I'm in control again, I like this idea." Look, RedMonk we generally... We don't like to make up terminology. We think the industry is better at doing that. That's not really our job. Some industry analysts do that. It's part of their business model. For us, that's not the case. We're like, we would much rather work with terms of art that exist. But sometimes it doesn't capture something, and there is fashion in this industry. And people wanna get excited about things and... Act accordingly. And so from my perspective, it was when it was like, "Okay, I'm having conversations with enterprises, they like this idea. It immediately resonates with them." But then also, when I look at the surrounding ecosystem of cloud and software companies, and they were like, "This makes perfect sense." So, it didn't matter whether I was talking to folks at like GitHub, GitLab is doing progressive delivery stuff now, HashiCorp were very much like Consul from a network perspective, definitely. They took on that, going back to my rage about service mesh, they were immediately like, "Ah yes, progressive delivery. This makes sense. This is something that we can drive into." Basically having customer conversations. The kind of thing that you're talking about. And I do think that, comfort factors are important. It's very easy if you are just a... You're a startup, you're a SaaS company, you're in San Francisco, you are already living in the future. But for everyone else, the comfort is just so, so important. And I think that's psychological safety, as one of the things that progressive delivery can bring is is exciting.

What also kind of gets me excited is, to your point, like being able to help some of these, kind of more established enterprise companies that have been around for a while. Really kind of have a path forward and like be able to see how they can actually make that transition from some type of... Product or service that they've, kind of associated their brand with, into this new kind of like digital age where they can actually compete, recognizing that they too, are now a software company and realize that it doesn't mean that they have to forego their relationship with a customer. And being able to make sure that that is something that they can continue to put first.

One of the things that become very clear to me, in all of my analysis, frankly, looking at software delivery as a discipline, looking at some of the topics around observability, how do you close the loop? Why do you close the loop? One of the things that become very clear to me actually, is that this is about the transition from IT projects to IT products. And so that sort of transition from projects to products, and product management focus, is where a lot of these disciplines really beginning to make sense. When you've got a small team led by a product owner, they're in charge of all of the services, all of the underpinnings that deliver that service to the end-user. And that creates I think, a different perspective on things. So I mean, if we look at organizations like the Financial Times now, they've got product donors. And in fact, if there is any underlying infrastructure that is not being used, or they don't know who it belongs to, they'll shut that off. And clearly in traditional IT if something's trundling along, you wouldn't switch it off.

You'd leave it alone.

Yeah, just leave it, right? If it's in a cupboard, you don't know what it is, don't touch it. It might be the databases running something important. So they've got this focus on like product managers that like literally, if the product manager doesn't own it, it has to be... If it's orphaned, it has to go. And I think that's the shift we're seeing to product owners, digital product owners. And that's the kind of transition that we have to help customers and enterprises to make because obviously that's where we're going and that's where the competition is. You're gonna be competing on the basis of digital products.

Well, it's interesting 'cause it reminds me of early days at VMware. A short, tangential story that, one of the primary use cases for VMware to get into financial services as a vertical, was the idea that folks had things running, that they didn't know how to migrate, because the person that had written it had since retired, and they didn't have anyone that was qualified to go and dig through the code. And so the number one feature or product that VMware introduced was called P2V which was physical to virtual.

And it was this idea that you could take a running service or system, you could actually point at, that system from kind of a secondary machine, and you could actually suck the entire image, including the memory state down into a virtual machine, and then bring it up and run it side by side to make sure that it actually continued to go. And so you could maintain and preserve systems that were completely out of date. We gave controls within the virtual machine to be able to make it so that you could actually run something that was built on NT4 on a new Windows 2000 server, because if you just tried to run the native application and do that transition, oftentimes you ran into race conditions because it actually was... The processor was too fast for some type of loop or some type of issue within the software. And this was something that, people were so risk-averse, that they were so concerned about the reduction in service, that these were the techniques they were using to be able to kind of move things forward, right?

It's another VMware story. I mean, people used to be afraid of automation. VMware was offering its customers like we'll automate this change If certain conditions happen, we'll do the change for you. And customers were not ready for that. So we had to have the slow customer button in effect.

Funny you mentioned it as a slowdown button. Internally... We talked about it as the level of annoyance that someone was willing to deal with. Because--

Yeah, yeah, yeah.

When you turn it all the way down, instead of actually just doing the automation, it would send a notification, "Hey, you should look at this automation." The problem was is that folks would be getting spammed with email or pages depending on how they sent that up multiple times a minute because that was how the frequency that the computer was recognizing that adjustments could be made. And so it was a really effective tool to be able to get people to trust it because they didn't want the notifications anymore.

God, (chuckling) notifications are the worst thing ever. I'm just gonna... I'm sure it'll be fine. I'm just gonna automate it.

It'll be fine. I'm sure it'll be fine. Yeah, it's funny that I think one of the things that we've been looking at, as we're now starting to see some of our more established customers that are looking to, go to the next level of where they can take feature management, we're starting to see this kind of emergence that. John Kodumal talked about earlier in the conference of this need for more end-to-end workflows. And so this is why we came up with the feature workflows functionality that we've launched and started to kind of roll out in the platform, is this ability to start to tie things together a little bit better for people. Be a little bit more prescriptive. Give people the ability to say, hey look, I know that I'm going to be doing a small feature and that has this certain rollout process that I wanna be able to have and versus like a big huge launch, that is gonna have more sophisticated process. Maybe with some approvals in place, some kind of scheduling or other types of metric checks that we're gonna be wanting to look at, all kind of flowing in and out of the LaunchDarkly platform, right? Because we definitely recognize, as we've talked about earlier, this notion that we still wanna think about it from a perspective of how are we giving the business control? But how are we doing in a way that like, enables engineering to continue just to move forward?

I think what you're saying is exactly right. And it gets to a couple of things. One is this look, the future is digital product management. That's not where we are now, in the enterprise in most cases. So that is a sort of a future state. But in order the enterprises are... That are able to work in that way, they're gonna need the right tools. And they are gonna need the right workflows. So workflows around features for product managers, that's interesting. Like, frankly, I think that's one of the big, big, untapped areas. It's better products for product managers, right? At the moment, we basically got like Excel and JIRA, and that's our product management software. So I think it's a whole area of opportunity to put better tools in those folks hands. If you're gonna do... How do you review features with your end users, identification of cohort, is that something the developer has to set up, or are there specialist roles in doing this. So I think the automation is important. And also one of the things that we've been thinking about at RedMonk, is look. It's very fashionable. People are "Oh, DevOps" it's a culture change, right? You can go and see a bunch of tech talks. People stand up and the first thing "Oh, DevOps is a culture change anyone else that tells you otherwise is selling something." I tend to think some of these people are selling culture change. But, that aside, the truth is, I think that there's an interplay between products, and workflows and culture. And tools can drive culture change, and culture change can drive tool change. So the two inform each other. And that's true in DevOps. It's true in things like DevSecOps. Sort of understanding that interplay. I mean, are we really gonna say that the sort of get-based version control systems, made social in the forms of like, GitHub, GitLab? Are we really gonna say that doesn't change how people work?

Right.

Clearly, there's been a substantive change in how people work. Tools can drive change, culture change, and vice versa. And I guess from your perspective, it'll be interesting to see the degree to which LaunchDarkly can become a platform, an opinionated platform, that could help to enable that culture change and transformation within your customers.

And I think that one of the things that's been exciting just since this summer is, we finally, kind of felt like it was time to actually launch our formal partner program. And the main motivation for this was, we got to a point where we'd been able to expose enough of our platform and do so in a way that enabled ecosystem partners, to be able to quickly and easily build integrations. To make it so that our joint users, were capable of being able to work better together between our platform and the other vendor, and make it so that it was a seamless exercise. And ultimately, this is what it comes down to. Of being able to think about this notion of automation of not only how do you automate things within your own services and systems, but how do you actually help to navigate that notion of tying things together from an input perspective. So what's coming to your system, as well as from an output perspective of what is your system actually able to push out to other services to continue that notion of workflow or usability? And so this has been something that's been really exciting to see how quickly we've been able to have folks step in and say, yes, no. We're hearing from our customers that... Our LaunchDarkly customers that, they wanna be able to have a more seamless solution. And we're hearing from folks, our customers themselves that are saying, this is a tremendous benefit to be able to incorporate other aspects of whether it's analytics, or monitoring and observability.

Right.

We've also done some great integrations with folks in the product management space, like Pendo that have been able to say, "Hey look, this is helping us enable the business and the business owners to be able to have that window back into engineering." That is an additive value as opposed to something that is disruption or an impediment to progress.

Yeah, I think that's awesome. The part you're playing especially I mean, look you're never gonna get a singleton in tools, right? Like, there's always gonna be a messy plethora of tools to be integrated and workflows accordingly. And so that that partner, that partner work, I think is just super important in terms of where you go in... You want an integrated system but you also need to be able to integrate with third party systems.

Yeah, so I mean, we're excited. I mean, I think that we're looking forward to kind of where things are headed. I find it really interesting that, in this global condition that we're all working through, we have an interesting opportunity to kind of see some acceleration in this area, where folks are saying, "Okay, well, I was sitting on the fence before, and now I don't really have an option."

No. Oh no, we've seen that in every organization... Every enterprise that we talked to. I mean, I think it's a couple of things. One is, frankly, think about a tug of war. There was a tug of war between people that wanted to move to the future and people that were happy with the way things were. I think that 2020 has been a significant slippage in the tug of war, when suddenly the feet lose traction in the mud. Well, the sticks in the mud have really lost traction. And we're not talking to any organizations that aren't really thinking about, how do we accelerate? How do we do more digitally? How do we get more into the cloud? I mean, we've had hair-raising stories. Automotive companies, where their testing is in the factory. So I mean, digital testing for software, but in machines that they have on-prem in their factory premises, without any cloud access, and literally they had to stop testing. I mean, this was an automotive company in Germany. And so now their entire testing infrastructure is gonna go into the cloud. And, look at any industry sector and you're gonna see a lot of that. So I think that, yeah, 2020 is definitely a forcing factor for pretty significant, substantive change. Another media company, a good friend of mine, very senior in that company. And he been, softly, softly, I don't wanna upset people, I would like to hire more people that are ready for a modern, fully automated, infrastructurous code, do all the things approach, but there are a lot of other people. And he's just, I'm not doing it. I can't afford to do that politics anymore. Because frankly, the financials of where we are as a business, aren't gonna allow it. It's time to move and we need to move fast and we're gonna move now. And that's just I think, across the board. And so moving to the sorts of approaches that we're talking about, yeah, I think there is and will be a significant acceleration in that regard.

And James, I wanna say thank you very much for joining me from the backyard.

Yep, thanks again.

Always a pleasure. And for everyone else that joined us for Trajectory, I wanna say thank you very much for joining us. Hope that this conference, and this session has been interesting and enjoyable and informative. And we look forward to hearing more of your stories. So we look forward to meeting more of you once we can actually get back on the road again, and hopefully seeing you around. So thank you very much.

Okay, thanks Adam.

Thanks, everybody. (upbeat music)