To Be Continuous: Effective Team Structures w/ Special Guest Kris Gale
In this episode of To Be Continuous, Edith and Paul are joined by Kris Gale, Co-Founder and CTO at Clover Health. They talk about what effective team structure for Continuous Delivery looks like. This is episode #7 in the To Be Continuous podcast series all about continuous delivery and software development.
This episode of To Be Continuous, brought to you by Heavybit and hosted by Paul Biggar of CircleCI and Edith Harbaugh of LaunchDarkly. To learn more about Heavybit, visit heavybit.com. While you’re there, check out their library, home to great educational talks from other developer company founders and industry leaders.
Edith: So, Kris, what do you like best about continuous delivery?
Kris: So, I like continuous delivery, because I think it’s sort of the first step of building a learning organization. It’s sort of a fundamental capability that a team needs to be able to learn collectively about what it’s doing in the world. The first step is being able to sort of ship increments of value continuously.
The second, I think, is to be able to measure the result of any incremental change you’ve made, so you can back out of a bad path that you’re on, or sort of refine what you’re doing and know that quantitatively.
The next step then, is to move from a sort of historical corporate planning, so this is a our six-month milestone, our two-year roadmap, our marketing commitments for Q3 of next year, into sort of an assumption that if we are shipping continuously and learning continuously because we’re measuring what we’re doing, whether it’s quantitative or qualitative measurements.
The next step is really being able to assume that in your planning, and know you’re going to have more information in the future than you had in the past. So your plans that you made six months ago are necessarily lacking information than plans you would make today would have, so sort of institutionalize that.
And then kind of the last part, and this is all kind of enabled by continuous delivery, the last part is building organizational fluidity.
So as an organization, whether it’s your project planning structure, or your accountability to your work flows, are fluid, so if you learn something new, you’re not locked into, oh we have a sixth person, I don’t know, sign-up flow team, and we just discovered that we have no levers for moving engagement right now in the sign-up flows, so we’re committed to that and we’ve got this implicit prioritization that’s built into our org chart. So, to have organizational fluidity where you’re adapting the organization to what you’re learning continuously.
Edith: Oh. You know that’s fascinating. Paul and I have talked a lot about, I think, more the mechanics of continuous delivery. We put it as kind of the last step to software org, when you’re really talking about how that changes entire organization beyond software. So, this would be a great time for you to introduce yourself, and tell us more about you.
Kris: Yeah, so I’m Kris Gale. Most recently, I was the VP of Engineering at Yammer. I heard you talking about Yammer in a previous episode. I’d love to dig into any of that, if you’ve got questions. Currently, though, I’m the co-founder of a company called Clover Health. So, what we’re actually doing is offering health insurance for seniors, with the idea that we can fill gaps in care, and actually prevent hospitalizations.
We’re not trying to be people’s doctors, but we’re trying to look across their consumption of health care and say, “Oh, here’s an opportunity ‘to inflect the probability with which ‘this population of people will be admitted to the hospital.” So, for example, making sure they get treatments regularly, get follow-up visits with specialists, that particular bio-markers don’t deviate outside of a specific range. And all this stuff is really interesting to me, because the health care industry as a whole doesn’t change what it does continuously.
We’re sort of locked into what people learned in medical school, or what people learned on their first day on the job at, say a big health insurance company, or something like that. But, to be able to power workflow with software, you’re able to continuously iterate and continuously refine and optimize the intervention protocols that are leading to these prevented diseases.
Edith: Yeah, I mean, I heard that you were pretty good at data loops, and that this is just basically using data loops for good, I think is how you say it.
Kris: See, the way we were talking about data loops, we were actually talking a little bit before we walked up here, is this idea that there’s something you intend to do. There’s data that you collect in doing it. There’s your objective assessment of that data to determine whether you’ve added or removed value or improved something.
And then closing the feedback loop, to get back now to actually modifying what you’re doing, and form find out new data. So, that’s sort of the data feedback loop, is from doing something to measuring it and deciding to do something differently the next time.
The tighter you can close that feedback loop, the more opportunities you have to optimize in a short amount of time.
Paul: So, it’s kind of what you might call the shipping loop, if you implicitly believe that data is a key part of shipping.
Kris: Yeah, yeah, and I would say that I think I do.
Edith: No arguments here.
Paul: Well, I think it is interesting, because I think that a lot of people don’t actually have measurements as part of their feedback loop, and especially, I think this especially happens in a small organization, where you have a start-up that sort of comes out of a founder’s vision, or something like that.
And so, one of the questions that I’m very interested about is, you were doing a big thing, with lots and lots of people. And now you’re doing a very small thing with, I presume, a small handful of people. How much of what you’ve learned still applies at that very small scale?
So, yeah, a lot of the project workflow strategy that we came up at Yammer was really about how do you maintain that agility, that velocity of value delivery, and ability to react to new information, and not be locked into, “Well, this is just the way we do it,” in the face of new information. At scale.
So, Yammer’s whole process was really about, how do you take these advantages we had, when there were like four or five of us in the really early days, and not have that big bureaucratic company, “Oh, we know we should be doing something better, “but we’re not,” because of momentum, inertia, bureaucracy, all these things. Like, how to break that. So, what’s cool about starting over with a much smaller team is, we’re just there again. We’re back at that five person size.
Edith: You’re a lot bigger than five people.
Kris: We are now, yeah. So, like nine months ago, we were. We’re like 65 people now, already. It keeps growing really, really fast. It’s been really helpful to me to have been through the process of scaling a software engineering team and seeing what that looks like. So, it’s like, “Oh, I’ve seen this problem before, “not worried about this.”
Paul: So, when you started, did you immediately set out organizational goals, and mission, and values, and that sort of thing that would help the company scale it?
Kris: Yeah, no, that’s a great question. I’m obviously a really big believer in the way we approached this problem space at Yammer, but I didn’t want anything to be dogmatic. I didn’t want anything to be–
Paul: You wanted it to grow organically.
Kris: Yeah, I wanted it to grow organically, to a certain point, but there have definitely been times where I stepped in and I said, “Nope, I’m buying a physical board “to track our projects, “because, just the way we’ve been doing this “with sticky notes, “I know that it needs this other dimension “of information capture. “I know that we’re suffering because it’s not there.
“So, I’m gonna actually pull the boss card and do this.” There have only been like three or four times I’ve done that in the last year, but it’s things where, like– Okay, I’ve seen where this goes off the rails, if we don’t address this right now.
Paul: And is that a result of knowing, sort of directionally, how the company should be doing things, or setting things up in some way that you didn’t get, that you didn’t end up in these bad places?
Or was it all sort of retroactive, and you know, a couple places where you pulled the boss card?
Kris: Yeah, it was mostly, like, observing patterns, saying, “Okay, we made this mistake three or four times, “and I’m noticing that our process or our approach here “is divergent from a known good approach “that I’m aware of.
“So, let’s at least–” I still wanna give the team full empowerment to iterate and own the process themselves, but let’s at least just pull the boss card, get it back on the rails, and then say, “Okay, let’s see how this works.” Just give everybody a chance to work in a way that I know to be okay for a few iterations, and then give them the keys again, and say, “Okay, take this where you want.”
Edith: Also, Kris, what I’m dying to know now is, what are those three or four key things where you gotta play that card?
Kris: Yeah, so, one is single project focus. Really, really important to me for software engineering teams. So, what I mean by that is, if you are on a project, that is your only accountability. You are not being split between three deliverables.
The other is that everybody work in cross-functional teams. So, that means not giving, for example, I might tell you, “Build out the back end for this feature, and I’ll do the front end.” So, in order for us to do that, we would have to agree to the abstractions ahead of time.
And again, I’m pretty opposed to planning, and locking in planning, because the information that we have now, when we’re trying to lock in those abstractions, is going to necessarily be incomplete.
Edith: I love that quote, because it’s so true. I think you put it more, it’s like you always know more in the future. It seems so obvious.
Kris: Yeah, I was actually– So, we had Parker from Pivotal Labs over at the office yesterday, and he said that there was this saying they had a Pivotal, that I really, really liked. He said somebody there would always say, and I wish I could remember who it was, would always say,
“You will always make better decisions with more information, and you will always have more information in the future.”
This is so much more concise than any way I’ve ever said that. But I think it exactly captures it.
But, on the other hand, my old boss, Gregg Brockway the founder of TripIt, he always said, “A good enough plan today is far superior to a perfect plan tomorrow.”
Edith: He’s like, “You just gotta start moving.”
Kris: Yeah, but I think the key is not locking yourself into that “good enough” plan. To allow yourself the mutability of changing that plan over time, so you’re not constrained by that.
Edith: Yeah, but that was TripIt. I mean, I think he’d seen so many, because he came out of the travel industry, where people just got paralyzed. Like completely paralyzed. “Oh, we need to plan for five years, “and what business travel will be like then.” It’s like, well, nobody knows.
Paul: I think the kind of key point to that is that in order to get that information, you have to get moving today with whatever information you have available. Otherwise, you’re just not gonna have that future information. You’re gonna be in the same place.
Edith: Well, look, sometimes, though, it’s painful. Sometimes you move backwards, then. Like, you don’t always move forward. Sometimes you start moving, and when you’re looking back, you’re like, “Actually we took, like, three steps back “before we started moving forward.”
Kris: It can definitely happen.
So to complete that thought, when we assign teams, we would always assign a cross-functional team with a business objective. So, it would never be I give you a back-end task, and I take the front-end task. It would be collectively, let’s accomplish this business goal.
So hopefully, we’re iterating day by day on the abstractions. So, we are collectively accountable to one another, and to this business goal we’re trying to deliver. Not onto some specific set of tasks that we broke off when we first decided to assign out the work.
So, that’s one of the other ones, is always cross-functional teams. Always single project focus within those.
Paul: So, there’s an interesting thing that you’re talking about there. It’s not just cross-functional teams, but it’s also that those cross-functional teams get put together for a specific project, is that right?
Kris: That’s absolutely right.
Paul: So, are those teams that will have worked together in the past? Is there explicitly trying to make sure that the people know how to work together?
Kris: Yeah, so this is one of my other core principles that I am pretty, pretty set on, which is rotation, is this idea that cross-functional teams are rotational and ephemeral. So, that would mean that maybe the three of us come together to ship a feature.
We’re all within the same organization, but when we’re done shipping that feature, we’re gonna break up.
You might be the back-end person that goes onto another iteration of this feature area, or maybe, you know, the two of us work on some other feature. Maybe it’s three completely different people. But constantly rotating people around, I think is really important in an organization.
I have totally heard the arguments about, you know, you can build high trust teams if they’re stable and stay together. But I’m trying to optimize for trust across the entire organization. So, across Yammer, we had 220 people in engineering. They would all rotate through these two to 10-person cross-functional teams together.
Which meant that you, basically within three to six months, would work with a pretty significant portion of those 220 people, and would start to build trust across the larger organization. Whereas had we just torn off four people and said, “You’re a team,” they would– And I’ve seen this throughout my career in engineering, where it’s like, “We do everything right, and that team has no idea what they’re doing.”
Edith: It’s their API’s are bad.
Kris: Yeah, yeah they use this weird language, or this approach, or we use this style guide, or whatever. So, it sort of forced that rotation, so that we’re all sharing best practices among one another and building, maybe lower trust, maybe not getting to that very, very, very highest level of trust among a group of four people, but across the entire organization, getting to like 40% or 60% of what’s possible versus 0% if you don’t ever work with those people.
Paul: So, interested in structurally how that ends up, because one of the things you got when you split people up by team is you end up with a sort of microservices architecture.
And this team works on this repo, or this project, or whatever works on this service or this API. And when you have those teams that are forming and splitting up and they’re ephemeral, and it’s only a small period of time, architecturally, what does that do to your application or your architecture, you know, across your entire product?
Kris: Yeah, no, totally. It was actually very intentional that we ended up with that kind of architecture for working this way. For two years, we had a big monolithic Ruby on Rails code base, and for two years we said, “This is a big problem. We should break this up.”
And everyone agreed, and everybody was on board with breaking it up, and two years later, we had never done it, because we continued to work in such a way that supported that sort of architecture.
So, the key for us was to say, well one of, the– So, something to dig into here a little bit, is these cross-functional teams are made up of representatives of what we call functional teams. So, front-end engineering might be one.
Paul: Right, right.
Kris: We had one called Core Services, which I actually owned the service-oriented architecture that you’re describing. Or there’s little micro-services that collectively made up our service-oriented architecture.
Paul: And this is sort of a matrix style, where you’re a member of the team, but you’re also a member of the back-end team, or you’re a member of the front-end, or the core services, or whatever.
Kris: Yeah, so I might be a member of core services. My manager is one of the core services managers. But all the work I do is going to be within the context of the cross-functional team, with the front-end engineer, maybe a client’s engineer, whoever’s necessary to get end-to-end with a value proposition.
And then, I’m always going to be a core services member, when I go into these cross-functional things, but it’s the fact that all the members belong to functional teams that makes that team cross-functional.
Paul: Got you. Spotify has a fascinating article or a case study about this.
Kris: Yeah. There’s a paper on this.
Paul: Yes. I think there’s like 4 different dimensions of their cross functional teams. You might be on the front-end team and then you’re on some sort of guild that’s interested in a particular kind of area maybe designer or some particular part of that. Then you’re actually on the team. There’s another layer that escapes me right now.
Kris: I don’t remember all of the details but it’s a similar concept.
Edith: It reminds me a lot of being a consultant.
Kris: Yes. If we’re in this room with a bunch of people actually the conversation I was having with partner from Pivotal was that this is actually very similar to how they did what they did which was consulting.
Edith: Well, because consulting … I was a consultant and it’s very much based on a customer. You are a body shop. Customer’s going to pay $200 an hour for people that meet their needs. It’s kind of like how do we assemble the A team of a front end and a back end, a product manager, a designer, to suit their needs. It’s very mercenary but yet it works.
Kris: Mercenary in … Do you mean that in like a pejorative way?
Edith: No, I mean it in a very positive way. I think what you were talking about how some companies start to have silos where you said like, “We’re the 6 person sign up team.” We know sign ups so we’re going to protect the sign up turf whether or not anybody really needs sign ups anymore versus what is the overall need of the customer?
Kris: Right. One of the things we were optimizing for it to really high level was to ensure that all of our engineering cycles were going to efforts that the whole company would agree is our top and most important efforts.
Edith: Not I like working on sign ups but …
Kris: Right. Yeah. We’re constantly fluidly right we prioritizing. Going back to those sort of 4 phases which by the way is stolen this guy Nate Fink who worked with us at Yammer. He actually took over my job there and …
Edith: You never steal anything. You just popularize ideas.
Kris: This is completely stolen. His framework and these 4 ideas. I think it’s just a great way to put it. The fourth is this organizational fluidity that allows you to reallocate staff to the things that are most important based on what you’ve most recently learned. It was really to us that if our CEO walked into the room and said, “What are the engineers doing?” We can point at a single document or in our case it was like a physical project board and be able to point to the 10 projects that were staffed, all the names of people and be able to say, “We’re only doing these 10 things because we have single project focus.” We will do these in a small iteration and then after that we can reprioritize. A week from now some people will be rolling off 1 of this 10 and we can say, “Okay. What’s the next most important thing for us to put resources towards?
Paul: It sounds like you always completed the projects once they started. Is that right? It sounds like there wasn’t a great deal of turn or just like this project is now important so we’re going to kill this one and move this one on the … It sounds like people actually finish.
Kris: I wouldn’t say we always finish it. If during the course of doing the project we discovered, “Oh. This is …” For example, we would not say this is a 5 week project. Do 5 weeks of coding and then ship it at the end. When it was working the best and I think the best tech leads that were in these projects were able to do incremental validation throughout the 5 weeks. It might be 3 days in where Dark Launching a feature and we know that … That’s right. Happy that I said that?
Edith: Thanks Kris.
Kris: A couple of days in we may Dark Launch an early, early iteration of it. The product manager can for the first time actually click it live with live data and maybe there’s no CSS on it, maybe it’s just like in a really sloppy form, but we’ll be able to see like, “Okay. Is this something that we want to continue to invest in?” If no or if we do an release of the feature to a small size of our users and then the metrics are really bad, we may just kill it without getting to the end of what was our initially spec as the feature.
Edith: Yeah. I mean a real issue we have a TripIt was so we had the same sort of structure where we had we called “The most important tasks lists” which was like the projects. It was always about 30 projects of which the top 7 actually people on them. They would happen is if you were on the top 7, you didn’t want to let your people go. This project’s like tier thing. How do you know when a project is done? It’s like well, I have my engineers. I’m not giving them up.
Paul: How are projects specified? It sounds like they were just very long-term projects that didn’t die?
Edith: No. They were meant to be short so there would be something like improved sign up flow and then let’s pick on that one. That would bubble up to like the number 3 project and them whoever was the PM would just be like, “Oh, I have people. I’m going to just keep going.” How would you make sure that these teams were short lived?
Kris: Yeah. One of the requirements of the products spec that would be the initial seed of a cross functional project was that that project spec have a definitive and finite done whatever that is. Maybe it’s like we come with a hypothesis that if we make this tweak to I don’t know message a replies will drive more engagement. Done is like when you’ve made that tweak and either confirm or disconfirm that hypothesis. Not keep flailing around and until you find something that works. That was really key for us is to agree ahead of time to what done is.
Paul: One of the things I like about Project Based Teams is the idea that the Project Based Team can keep an eye on the parties within that thing. We’re going to fix this very, very small bug, a CCS alignment sort of thing or focus our efforts on a monumental rewrite. It sounds like you got the monumental rewrite taken care of. It’s obvious how that works. How do you take care of the we shipped it and it’s kind of like 95% perfect so we’ll call this done. There’s all these small tasks …
Edith: That’s what I meant what happened at TripIt and that’s why people clung.
Paul: Right. How is it handled at Yammer?
Kris: Yeah. You know it’s interesting. I think actually in complete honesty, I think we were not great about polish. I think there were some things that some teams would just polish and polish and polish other companies in a way that they produced a more polished result than what we did. I think there was a potentially logically coherent and rational argument that if we couldn’t measure the impact of that polish because there was no way to quantify that it had value, maybe we were actually doing the right things. Maybe the right thing to do was instead of doing the remaining 4, 5% of this project that’s going to make you feel really, really slick, maybe putting those engineering resources into starting to validate or invalidate some other hypothesis in some area. It’s going to deliver more business value faster. I would argue and I’m not saying that I know that for sure –
Kris: – that’s sort of the position that we took. Though on some features where polish was important, the definition of done could include that polish. It was fair for a PM to say, “No, no, no. This isn’t done until this is right in these ways and these pixels are lined up in this way. This all works. This edge case has to be dealt with. That’s part of the spec.”
Paul: I guess the thing that I’m interested in is where the final results come out of. Let’s this put and wait for 3 weeks worth of user data or feedback or whatever. What happens to the team? Where does the polish get actually done? Is it in new project later down the line or how does that get handled?
Kris: Yeah. My favorite way of addressing this was really just to elect the PM team. Not keep a backlog and I’m actually pretty opposed to like formalize backlogs we were talking about a little bit earlier. To have a general idea of the next time we revisit this feature area, I’d really like to clean up all these things.
Edith: We’ll actually really love to hear you why you’re against the backlog.
Kris: Backlogs are designed inventory. We’re talking about the definition of lean. I think lean at least in my mental model of it is about minimizing inventory.
Kris: I think if you have a backlog you have to committed to a bunch of design inventory. You’re saying that at some future point we’re going to do all this stuff. One thing that’s nice about not maintaining a backlog is you can actually choose never to revisit something. For example on this some of these polish questions, you might say, “Well, this interface … It would be really nice if we fix this thing or whatever.” Then you do an iteration that just blows away that whole feature are. You never actually have to do that.
Edith: That’s interesting. I sometimes like maintaining a backlog just to acknowledge that we’re explicitly not doing it. That’s kind of an anti-requirement. That in every meeting somebody’s … Because somebody’s always going to pipe up, “Why don’t we do this?” Like, “That’s in the back log.”
Paul: Right. In organizations that use bug trackers or where their entire thing goes into bug track because it’s very useful to have the bug that is marked as won’t fix so that you can always just refer to people we decided not to fix this for some reason.
Edith: Because otherwise you could spend like half an hour in a mini rat holing on them, “Why are we doing this?”
Paul: Right. You can start coming up with the same ideas again and again because you’ve blown away the institutional knowledge that it comes up from having that recorded somewhere.
Edith: We’ve acknowledged that this issue exists and we’ve decided that we’re not going to do anything.
Paul: You’re looking sort of cynical on this.
Kris: I almost think like deciding not to do something is potentially as harmful in that sort of like we’ve made a long-term plan. It’s deciding to do something.
Edith: This is why you’re a great guest Kris.
Kris: It’s just that argument.
Edith: Of Course Kris has a new perspective.
Kris: I like the rebubbling of things for reconsideration later when we know more. It’s like, “Yeah. We decided not to do that but we decided to that with incomplete information in the past.”
Edith: That’s fair.
Kris: “In the future we may have learned that some other feature area, this was actually really significant or something that looks like this feature was meaningful for our core metrics.” Now the next time this bubble’s up, we’ve got a fresh perspective on it. We can reconsider, “Okay. Are we going to do that or not?”
Paul: One of the things I’ve seen that comes up with this is the sort of a constant back and forth between 2 non-optimal options. Say for example we can combine these things and we can separate these things. When they’re separated, it really looks like they should be combined. When they’re combined, it really looks like they should be separated. If you lose the institutional knowledge of this is why we combined them the last time, then you often end up with a whole load of like just spinning your wheels, getting nowhere on that. How do you avoid that?
Kris: I think … Yeah.
Paul: I guess you’re not saying we’re just going to throw away all the institutional knowledge.
Kris: This is another contrary position that I like to take which is I love losing institutional knowledge.
Edith: Wait. Kris, you’re the best guest ever. You’re also our first guest but …
Kris: The most ridiculous one. I have some really good examples of … I’m actually probably less informed about it. The question you’re asking is really about product decisions. Like product design decisions, UI decisions?
Paul: I think it applies both to technical decisions.
Kris: Okay. Let’s talk about the technical decisions. This is where I feel really passionate.
Kris: There were a bunch of cases. I built most of the original messaging infrastructure at Yammer. I have strong opinions as a software engineer. There were parts of the code that I’d be like, “Hey. If you’re ever in there, ask me about it. There’s a specific reason it’s like that. I can justify all this. I would document it or there’d be really robust code comments. It said don’t touch this because it’s like this because of this.” Actually what I found was over time that super disempowering for future people. They don’t own the architecture decision. Some past person who wrote something down is dictating how they do their job now.
Paul: Not the company anymore and you can’t actually talk to them.
Kris: Maybe their now in some management role and they’re not available in that moment when you need to change the code because they’re off doing a one-on-one with somebody about something really important or talking to HR doing something. You’ve disempowered the engineers to do stuff. A lot of times what would happen is institutional knowledge often defends complexity. I really, really like losing complexity. I would rather people go to the code and say, “I don’t understand what’s going on here. All the test past when I stripped out this thing. I don’t know. Maybe I broke something. There was no test for it. Maybe it’s a weird edge case that just never comes up.” Let that bubble up again. Let the code shed the complexity.
It’s easier to maintain, easier to understand, easier to like into about and fix that bug if it comes up in the future. What really happens 99% of the time is by not having that institutional knowledge, the codes are simpler and that weird edge case that came up that 1 time that you wrote an elegant solution for, is actually just slowing everyone down and you’re better off for not having it.
Edith: This is basically the buffalo herd of code. You know the buffalo herd? There’s a herd of buffalos, and the theory is you kill the weaker and the herd moves faster.
Paul: Okay. The opposing view and I think something that most of my generation who have been educated with is this … Jules Feiffer wrote this thing about the Netscape 4 rewriters or something where they just tossed it all away and then they had to rewrite the FTP module and they had to integrate with the 50 kinds of FTP service and no one had that technical knowledge, institutional knowledge whatever it is you’ll call it. I’m feeling some sort of like level of anxiety based on what you’re saying there. Yeah. It seems intuitively wrong.
Kris: I think if you are forced to stop and rewrite and entire code base, you’ve done something wrong upstream. Like yeah, we can try to optimize for that case for ..
Paul: Sure. Sure. I mean what you’re saying is we can take this module or this small thing and we can just discard the information that was involved in building it.
Kris: What I’m talking about doing is getting to a situation where everyone is empowered to make fresh decisions to continuously re-factor and continuously modify the code without being disempowered by previous decisions. Potentially yes, trading the fact that we may temporarily lose some functionality and maybe the way to reintroduce it actually ends up being cleaner because now we’re free to do that re-factor or that cleanup or the simplication without …. That institutional knowledge is just so, so often used to like, “Don’t touch that because … Don’t change that because … We can’t move on to this new framework because …” I never want those things. I would rather make the little mistakes and come back.
Edith: I think a key part of this having a really good measurement system. I think Paul’s fear comes from like not just simply cross your code pace and then toss everything back out but have a lot of … At Yammer, you had a lot of measurement. You had a lot of analytics. You knew that if you changed something and suddenly your conversion would down 20%, you even knew it.
Kris: Test are obviously important for this. A well-tested code base. The test are to a certain extent found in institutional knowledge. The code should do this in this case or the product needs to do this in this situation.
Paul: There’s then seem trade off there between … You’re talking about test as 1 way to fix this and Edith’s talking about measurements. It occurs to me that tests are a sort of binary version of a measurement. The measurement is an actual business value whereas a test is a thing that someone wrote at some time in the past but that is black and white and irrevocable. You can’t break the test.
Kris: No, I think you’re right. I mean the measurements are essentially a form of test that just aren’t so binary. Did we move engagement? Did we break our conversion …
Paul: It seems like the test are much lower value than by definition because the business value is measured in these KPIs or whatever the things are that you’re measuring.
Edith: Yeah. I’d love to hear you talk a little bit more about Yammer’s measurement framework.
Kris: Yeah. Yammer really cared about engagement, which was our most important KPI. We were sort of banking on the fact that as a kind of bottoms up enterprise tool, that the more people that use it and the more often they use and the stickier they were with the product, the more likely their companies were to buy the product in the future. That was really … It’s pretty simple. It was like, “Does the user come back and continue to use the product if they’ve been exposed to this or if they’re in this group or if they have this functionality?” We obviously looked at other things lik