5 Real Ways to Use Feature Flags Across Product, Marketing, Sales, User Research and Legal

Heidi Waterhouse: So we've come back from lunch and now we get to hear a session that's going to be super cool from a first-time speaker. Emily works for Seesaw and they are an educational software company working to empower and enable both students and parent to understand better what's going on. I sort of think of it as instrumentation for homework. So Emily's going to talk to us about five ways they actually use feature flags at Seesaw and how that's been working out for them. So I'm super excited to hear this. Please give her a very warm welcome.

Emily Voigtlander: Hi everyone. All right, so let's get this going. I'm here to talk today about five ways that we used feature flags last year in our first year of using feature flags at our scaling startup. First, I'm Emily. I'm a product manager at Seesaw. Our mission at Seesaw is to create an environment where students are inspired to be their best. Seesaw is used by millions of students in more than half of the schools in the United States, and over 150 countries around the world. And students really use Seesaw to share their learning with the people who care most about their success. This means we have a bunch of creative tools that students can use to show their learning, and that learning gets shared with their teachers and their families so that families can be involved in the learning process and support their student every day instead of once or twice a year at a parent-teacher conference. And teachers really have a good understanding of how each student is doing in their class and how they can better support them as they're on this sort of educational journey together.

As a PM, my job really is to ship a great product to our students, teachers, families, and school administrators. And for anybody who's not a PM, another really big part of my job is bringing the rest of the company along for the ride. It's not just about building great stuff; it's about making sure that everybody internally and also externally understands why we're building what we're building and why it matters for our customers.

And so when I think back to where we stared, I joined Seesaw in 2014. We were a company of three people, and over time we grew to five people and that felt huge, right? It was like not just me and two co-founders talking in a single room at a single desk. We actually didn't have a desk; it was a couch. And then over the past few years the team has grown a lot. We've had the amazing opportunity to grow the team because we actually built a product that schools wanted to buy and give us money for, which meant we could grow our team. When we were really small, it was super easy for us to stay on the same page because we sat in one room, we could only work on one thing at a time, and we had lunch together every day.

Now it's harder. We have 50 people on our team working across product, engineering, sales, marketing, customer success, operations. We have departments now which is crazy, but that means that it's actually a job to keep everybody in the loop and make sure not only are we building the right product for our customers, but that everybody understands why we're moving in that direction and is able to communicate that product vision, and really have an impact on our customers, and using sort of their role and responsibilities to sort of channel that.

Another exciting piece of how the company sort of has changed as our customer base has scaled is now millions of students need Seesaw to just work for them. When they open the Seesaw app or go to the Seesaw website, Seesaw needs to be there and it needs to be working. When we had a few thousand classrooms using Seesaw, if an issue or a bug affected some of those customers the sphere of impact was also pretty small and so it didn't really matter as much. Now Seesaw is a tool in classrooms like GitHub or Slack is a tool for all of you. So if you opened your computer on Wednesday, and GitHub or Slack wasn't there and wasn't working or looked completely different than it did on Tuesday, you're going to have a problem with that and it's going to really impact your day. Imagine you're a teacher, you're teaching 30 students and those 30 kindergartners open Seesaw and Seesaw isn't working or looks different. Now you have 30 people. Imagine your entire team panicking about not being able to do their work.

Also now we have a product that people are really using and loving. Hooray. But we need to make sure that we can continue to grow and grow that product, change it, evolve it in the right ways, but also not totally disrupt our customers in the middle of the key sort of workflow for them. So we had to sort of reflect on that growth in the company and the growth in the customer base and really ask ourselves like,  "How do we build a great product and engage the right people in that process? "

And now sort of also we have more people on the team. We have more engineers. We have more capacity, the thing that we've been wanting right from the very beginning. Instead of just being five people, we now have 20 engineers. What does this mean? It means we can do more than one thing at a time, but that also means we can also break more than one thing at a time and so we want to make sure to work on more than one thing at a time and also have that not be a total disaster.

So this is where feature flags enter the picture. Hooray. In early 2018, at the strong pleading of one very vocal engineer, we added LaunchDarkly to our team's toolkit. And I have to say it has been a total game changer to the way we develop product at Seesaw. We do this thing where, at the end of the year, everyone has one slide where they can propose some change that they want to see in the company. This particular engineer was like,  "Feature flags. " And honestly, I had not really ... I didn't even really know what he meant when he said feature flags, and I think it's really changed the way that we do work at Seesaw. And so we're going to talk a little bit about what that looked like, specifically for the product that we're building.

So the first thing that we learned is don't forget you have feature flags. Adding something new to your team's workflow, even if you're a small team that is excited about changing, is still hard. You have to remember that you have this new tool in your toolkit. So one of the things that we did was just add a small reminder into our product and technical spec template. It's really just like a one-page document that's like,  "Are we making sure we're doing the right stuff? " We have other things that I've forgotten when writing various product specs or other people have forgotten when writing their technical specs in here like,  "Oh, we should probably figure out what we're going to do with notifications if this feature impacts that. " We added feature flags into here and have made feature flags sort of part of the conversation early in the product development process so that kind of everybody on the product and engineering side is sort of on the same page about how we're thinking about this feature and how we're thinking about flagging it.

So the actual ways we used feature flags. So I'm going to talk about the ways we used feature flags across a few areas of our business: product, marketing, sales, legal, and user research. And each of these teams have different goals. We use feature flags in different ways to support them and really make sure that they're along for the ride.

So our first ... on the product side, our goal was to launch a new feature on three platforms. Seesaw supports iOS, Android, and the web. We need to make sure that we launch the same features on all of these platforms at about the same time. At Seesaw specifically, our goal last year was to launch a lesson library and assignment flow that integrates with a bunch of our existing product but also had a bunch of new features. So we have got this tricky thing of saying,  "We have some new features that we want to add in while also integrating with existing product features. " So our teachers are going to find a lesson in the library, they're going to share that lesson to their students, students will respond using all of the stuff that we already have, and then teachers are going to review those responses before they're shared with families. So we're merging sort of new features, existing features. How's this going to work?

So we used flags a variety of ways on this project. Sort of the idea for the lesson library was originally inspired by behavior we saw in our community. Teachers were creating these slide decks of all of these lesson templates and sharing them on Twitter, and Facebook, and stuff like that. We're like,  "Great. We can take that. We can make something better. " And so in order to get this feedback on this feature early, we were able to reach out to those content creators and demo the features or have them test those features in sort of early beta of those features and get feedback that we incorporated into the product development and engineering process.

We also use flags to verify backwards compatibility and integration with our existing product features. Before feature flags, basically we would just kind of hope for the best here with integrating with the existing product features. We would test as much as we could in beta, and then we'd basically just like push everything to production, and then quickly test in production and make sure it wasn't terrible. Now with feature flags we can just turn the feature off, test, make sure everything is working, turn the feature on, test, make sure everything is working. Way, way simpler. In the same environment because there's always differences between production and beta, all this stuff. And as a result, we've had far fewer pretty bad bugs get out to our customers which is really important when you've got those 30 kindergartners who are expecting Seesaw to work for them.

Flags were also super helpful for us with synchronizing the rollout of all of our features on iOS, Android, and the web at the same time. We want to make sure ... you know, we have a native iOS app, a native Android app, and a web app, and a shared backend. We want to make sure that the features are available for everybody, but as many of you probably know, you need to submit your iOS app and get that one approved by Apple before it can go live to your customers. Android, you can kind of submit any time. And the web, we can push whenever we want.

Before feature flags we would submit iOS, and wait for the app to get approved, and then push web to production, and then quickly release iOS, and test to make sure it was working, and then if that was okay we'd submit the Android app. Now we can submit all of the apps and push to production on whatever schedule we want, whatever works for our development team. And then product can work with marketing to turn the flags on for a feature at a specific time. So basically instead of having a release day that was basically like a ton of chaos, we can just release features when we want to turn features on for the customers we want to turn them on for. It's amazing.

So I think feature flags is also one of the reasons this feature has been like the most quickly and widest adopted feature of any feature that we've launched since our initial release. We have launched many features since 2015, but this lesson library and assignment feature was the quickest adopted. Typically features will take almost a school year cycle for teachers to really incorporate them into their routine, and this one was adopted super, super quickly, and as I think one of our sort of popular features other than the first feature which was like the portfolio for student work. So really successful for us, and I think really do a lot to ... you know, the fact that the feature actually solved a real customer need and also worked in the classroom. And all of that was really possible because of these sort of different checkpoints along the way that were enabled by feature flags.

Cool. Okay. Next one. Marketing. Why does marketing want feature flags? Okay. So our goal for this project or sort of a generalizable goal that maybe any of you might have is to get user-generated content before a public release. So for us at Seesaw, this meant building out a library of thousands of teacher-created lesson ideas that are ready to use in Seesaw classrooms but before the feature is released. So before we had five Seesaw-made lessons. We controlled the content and there was very little of it. So what we want to say is we're going to go from this, five activities per grade level created internally to tens of thousands of teacher-made activities in the activity library that teachers can use. So how did flags help us with this?

First, we gave specific brand ambassadors and power-user teachers early access to this new feature flow and then also gave them the ability to contribute their lesson ideas to our library. Our goal really was that we would have some amount of high quality user-generated content in the lesson library before it was released publicly to all of our teachers in the community. We didn't want them to get there and basically have like ... it's like,  "We launched a lesson library feature and there's five things in it, " right? We wanted to say like,  "We launched a lesson library and there's thousands of lessons that you can use in your classroom tomorrow. " So because of feature flags, we were able to get I would say I think like 3,000 teacher-created lessons, which I honestly I'm not really sure how we would've done before feature flags. And now, post-release, this feature is public. It's available to all of our teachers.

We now sort of have on the marketing side a different goal. We've got a steady stream of content coming in. We have contributing authors sharing new ideas every week. But the problem is now how can we make sure that the lessons that we have in the library are high quality and meet the expectations that we have for sort of the curriculum that we're putting out into classrooms?

And so now, we sort of shifted gears and now use feature flags to allow contributions from teachers who meet specific criteria to maintain the quality of the submissions. So not every teacher who's using our app can submit, only teachers who meet specific criteria that the marketing team ... and the marketing team has flexibility to decide what that criteria is, which is also great because it means we don't need an engineer to add a user to a specific list to give them access to this special feature. The marketing team can decide like,  "We want to open up contributions to this grade level or this type of teacher customer, " and they have the power to do that which is super cool.

Sales. Okay. So the goal for this feature was basically to preview a top-requested feature to a high-value customer. At Seesaw specifically, we wanted to preview some new student data privacy controls to these high-value customers who had requested these features. We also wanted to get a little bit of in-the-wild testing in a contained sort of environment before we rolled out a control to all of our paying customers, right? We want to keep our best customers happy, and really involve them in the product development process, and also not break things for everybody else.

So for this particular feature, it's very technical and also impacts student privacy. So making sure that it works the way that school administrators expect is really important for the success of the feature. And to be honest, we weren't exactly sure we got a lot ... we work with a lot of different schools.

Each school thinks that they are the most unique, and in many ways they are, but we're really trying to build a product that's sort of a little bit more universal and able to be sold and applied to a lot of different school environments, public schools, charter schools, international schools. And so we're always looking for product solutions and engineering solutions that are flexible enough to handle the most amount of cases. But that also means it's not perfectly customized to every customer, so maybe not exactly what they expect. And that was definitely the case with this privacy feature.

So flags helped us a lot in this case. We made a bunch of changes on the backend to support this feature. We release those into production on our regular release schedule, and then only revealed the frontend UI, that dialogue that you saw, to specific customers and let sales work with those customers to get it properly set up for their account, talk them through how the feature works, how to set it up, and really sort of help those really important, our biggest customers, most amount of students be successful with this feature.

And then the flags also really helped us for this feature because it involved a lot of different sort of ... We were expecting behavior on the school network for the product to be different than behavior on students' home network. And transitioning, you know,  "You got a computer on the school network, and you bring it home, and is it doing the right stuff? " was really important, but also a little bit unpredictable. And so instead of releasing this feature to all customers, we really tested for a few weeks with these customers who asked for the feature, so validate that the customer is happy with the feature and also de-risk sort of rolling this out on a broader scale for all of our customers. And it was great. It worked. They were happy. Hooray!

User research. So the sort of user research goal that we were working on is basically validate that changes that we made to a key user flow didn't actually make that flow worse. So at Seesaw, we basically we do a lot of usability research. We learn some stuff in usability research. The product team and design team work to iterate on that flow. And our goal is really to make the teacher onboarding experience simpler and not more confusing, but we don't know for sure, right? Because we are so close to this onboarding flow, we're not really sure if it's actually helping, and so we worked with ...

So basically, this is where we started. We had this epic getting started checklist, and guess what? It didn't convert. If you used any of these things, any of them, including dismissing this checklist, you were just as likely to become a successful Seesaw teacher. Not great, right? And we originally started with this idea because we were like,  "What's the sort of quickest thing that we can build that we think will not be terrible? " And we definitely didn't build something terrible, but we also didn't build something successful. It had no impact on conversion.

So from watching a bunch of very painful user research studies where teachers told us all of the ways that our product was confusing. Very useful. Really recommend. We came up with a different approach to teacher onboarding, which was the tool tip walkthrough based on state. So we would look at the state of your account,  "Have you created a class? Have you added students? Have you shared a lesson with them? " and then prompt you to do these things based on the state of your account. Much more customized and hopefully more successful, right? But we weren't sure.

And in EdTech, we kind of have one really solid shot to get a bunch of new customers and that is back-to-school. So August and September are go time. We really need a product that we feel confident in in advance of that time. It's a critical growth opportunity for our business. Unlike other consumer apps where you can get new people all the time, we have a very specific season that we can make this happen.

And unfortunately, we also have a small team, and we're working fast, and we spent most of the year building that lesson library feature, and then we're like,  "Ah, we probably should handle that onboarding problem. " And so we didn't have a ton of time to A/B test things, evaluate the results, a lot of other things that folks have talked about in earlier presentations, which are all awesome ideas and definitely do them. But if you're short on time, you can do what we did, which is basically let our user researcher test this new flow with specific participants by turning this feature on just for them in production, and validate that those changes, though we don't know that they could make them ...

We can't measure yet how much better it made the onboarding flow. We could see pretty quickly made it disastrously works, like people were more confused than in previous studies. And in that case, we just wouldn't turn the feature on for customers. It turned out that it didn't make things worse, that people like a customized onboarding experience, and hopefully we can invest more in that teacher onboarding over time. But yeah, so flags really helped us test something that was going to impact a critical flow at a really important time of year and sort of de-risk that a little bit.

And then everybody's favorite: legal. All right. So goal: sort of comply with key privacy regulations. No matter what industry you're in, the data privacy of your customers is important. At Seesaw especially, we wanted to maintain our strong commitments to student privacy by complying with GDPR for our EU customers, and the Canadian Privacy Act for our Canadian customers. What's great for us is that we already had really strong privacy commitments before. We didn't sell student data. We had worked on our permanent deletion project earlier that year. We don't show ads. So complying with these privacy regulations was a little bit easier for us than maybe other companies because they were already in line with the philosophy of our company. But still, we had to do some work in order to make sure that we were compliant with these privacy laws.

So before, everybody got the same info. There's a link to the terms of service and privacy policy on the bottom of the sign-in screen, and that's basically it. Under GDPR and Canadian Privacy Act, just having a passive link like this is not enough. And so after, we want to make sure we're showing the right info to the right customers, right? We don't need to show the same information to all user roles. For example, we don't need to show this information to students but we do need to show it to teachers and school administrators. We want to show a slightly different message that's customized for family members signing up for the service. And so it's really important that we're showing the right information to the right customer at the right time, and feature flags was one way we were able to do that.

Testing matrix for testing this looked a little bit of something like this. We have basically tons of lines of all of the different cases of all of the different user roles for all of the different countries on all of the different devices, and then our different apps across the bottom. Email too, right? It's crazy. And so feature flags were super helpful for us in this context. To specifically stand out, first we created a country spoof feature flag. So different environments, developer environments, beta, QA, production, we could spoof specific countries. So we could say like,  "Okay, pretend I'm in the U.K. right now. What am I going to see when I sign up as a new user? Pretend I'm in the U.S. right now. What am I going to see? Pretend I'm in China right now. What am I going to see? "

The US right now. What am I going to see? Pretend I'm in China right now. What am I gonna see? And similarly, we use a date spoof flag to basically trick the app. We want to show these privacy regulations after GDPR goes into effect on May 24th or whatever, and so we can change the trick, the computer to think that the date is May 24th. What am I going to see if I sign up on May 23rd? What am I going to see? And so, making sure that we really handle showing the right information to the right role was really important. So flags were super useful on the testing side, make sure we're showing the right stuff, and then also allowed us to preview them with legal before the release. They had seen the mocks, but they want to see the real thing before we can officially say and tell all of our customers that we're compliant with GDPR.

My favorite. If you only remember one slide, add feature flags into your spec template and try to talk about them early across your entire product and engineering team. Flags are your flexibility. So if flags are new to your team like they were to us, over flag stuff to start. We started on the other side. We're only gonna use flags for big features. Big mistake. The more flags you have, the more flexibility you have and the more options that you have, no matter who you are in the organization. On product, on engineering, this gives us a ton of flexibility. If we make a mistake, which we definitely do, but we want to make sure we still have a really high quality product experience for our customers in production, we just turn off the flag or we change who it's targeting.

And then, this will mean you have a proliferation of feature flags. A lot of them. And so, you should probably plan to add time to clean up your feature flags. If that is not part of your development cycle because you're really trying to hit a deadline, we use the tags feature that LaunchDarkly has to organize flags by team or by project, and that's really helped us just stay organized and not lose track of all of our flags and know what's going on for what team and what environment.

Questions? Yeah?

Audience Member: [inaudible].

Emily: Thank you. Yeah. So the question was, did anybody on the non-engineering side have trouble adapting to this new process?

So for the sales and marketing and user research examples that we shared, the PM on the project worked pretty closely with our marketing team, sales team to learn how to use the features. I think that basically everybody was pretty excited because it meant that they could show stuff sooner to the customers that they care most about and get feedback earlier, and help us build and deliver a better product.

On the engineering side, basically everyone was excited because we were doing this crazy process before where we would have these long live branches like people have mentioned before, and that we would merge in at the last minute and then quickly test on beta and get them out. And it was just really brittle. And so, I think everybody on the engineering side was excited to make our process more robust.

Yeah.

Audience Member: [inaudible].

Emily: Yeah, totally. So we have a few development environments, and so typically if we were testing something that we were ... That we wanted to be able to test in a branch or conflict, we would push to a different environment, turn the flags on there for testing. But typically, we're trying to basically get everything merged into master, and so that's going to be what's in production for our customers. And so, those intersections or overlap, we need to actually find out about those in order to make sure we don't have that same problem in production.

I think it definitely ... Working on more than one thing at a time requires more communication between teams. I don't think feature flags are going to solve all of that. That would be amazing if they did, but humans are humans. And so, tech leads and PMs definitely spend time working with the other tech lead and PMs on different projects to make sure that we aren't going to collide in some horrific way. But so far, not too bad. Does that answer the question?

Audience Member: Yeah. [inaudible].

Emily: Yeah, that's totally happened. I think we've had experiences where we're like,  "The features are gone! " And then it's like,  "The flags are on in QA, but not in beta, " or something like that. I think the more flags that you have, the more likely it is that's gonna happen. We have tried to spend some time cleaning up flags that become permanent parts of the code base so that happens less. And I think really for us, this idea of over flagging was really just at the start and we've been able to hone that over the past year. So not every single commit is behind a feature flag. And that has also helped with the proliferation of flags, or complexity versus value of the flag.

But if we're doing something like upgrading our video player and we have tons of student work that's videos, let's put that behind a feature flag so that we don't have to revert that commit a few minutes before we push to production because we discovered that the video player doesn't work on IE or something like that. Yeah.

Audience Member: [inaudible].

Emily: Yeah. Right now legal and sales is coming to me. The marketing team has access to the flags, and obviously product and engineering, but we have two account managers and we just worked with them to say,  "What accounts do we want to turn this on for? You'll handle all the customer communication and we'll just get the feature turned on for you. " That's what we're doing right now. But the team's still pretty small.

Okay. Great. Well, thank you all for your time. I really appreciate it. And if you have any questions, my email. We are also hiring, so if you want to help me build great software for elementary school students, please reach out. I would love to talk to you. And thank you all so much.