Unlocking the Monolith: Lessons Learned from Naviance's Journey

In this talk, I will share Naviance’s experience using Launch Darkly to power our next generation of software architecture and team practices.  I’ll describe our team’s blueprint for progressively breaking up a monolithic architecture of roughly 1 million LOC  into a service-oriented architecture more accurately reflecting our application’s Bounded Contexts.  We’ll discuss how this “modernization blueprint” connects to our use of feature flags for controlled releases, but also look at other aspects where the importance of feature flags may not be readily apparent (for example, when performing data migrations).

The talk will be divided into five key sections. 

Section 1: Why?  Modernizing to Scale Your Team and Unlock Innovation

We’ll begin the discussion with a quick snapshot of the Naviance team’s situation prior to our use of Launch Darkly feature flags and our commitment to progressive, opportunistic modernization.  This section will cover a few key decisions:

1.1.  The Monolith Doesn’t Scale:  To get things started, I want to begin by describing the “good problem” of having a successful enterprise product.  For Naviance, this has meant YoY growth in our overall team size and our ability to bring new features to our customers.  However, this growth also poses a problem: a single codebase that can be maintained and enhanced successfully by a handful of teams simply cannot scale to concurrent work by hundreds of engineers.  

1.2  The Monolith “Locks In” Technology:  Similar to the prior sub-section, in this section I will describe how our team had a strong desire to migrate towards new technologies and patterns in order to take advantage of modern, high-velocity practices, such as Cloud Native architectures and Serverless designs

1.3   The Monolithic Release Cycle:  In this section, I will describe how our monolithic application also locked us in to a lengthy release cycle that complicated our engineering, increased our risk, and delayed our ability to release value to our customers.  At this point I will introduce Launch Darkly, and discuss how our team initially looked at Launch Darkly to solve this problem in isolation, only to realize later that feature flags also unlocked our ability to attack the prior two problems as well.

Section 2: How To Get Started: “Big Bang” vs. Opportunism

In this section, I will describe our team’s first key strategic decision: Should our existing monolith be rewritten from scratch or should it be progressively modernized?

In this section, I’ll bring our team’s experience to the forefront by highlighting that we’ve done it *both* ways: Naviance has rewritten a monolith of roughly 700k LOC from scratch in a new programming language (our student-facing application), and we’ve also progressively enhanced our school staff and teacher application by “Breaking apart the monolith” module-by-module. I’ll present a comparison matrix that shows the benefits and challenges that come from both approaches.  The key takeaway of this section will be that, while both options are better than nothing, opportunistic enhancement allows more freedom to bring added value to the customer and improve the overall system architecture.  

This section will also provide some commentary on how Modernization is more than a technical decision.  Committing to a strategy of progressive modernization requires a deep understanding of your customer’s needs in order to be confident that your technical and market vision are in alignment.

Lastly, this section will conclude with a “Big Picture” diagram of what a progressively modernized architecture can look like over time.

Section 3: “The Blueprint”

In this section, we’ll look at the architectural blueprint Naviance has replicated across multiple teams to migrate monolithic application modules into standalone services owned by small, individual teams.  At a high level, this section will cover the following list of key features of this blueprint:

• Feature Flags for Progressive Rollout
• “Micro Front Ends” (a la https://martinfowler.com/articles/micro-frontends.html) as a pattern to build modules that are truly loosely coupled from an existing monolithic system
• Data Migrations:  We’ll review an event-based approach for migrating data in two scenarios:
• “Initial Data Load”
• “Data Backfill” for reporting

This section will conclude with some key takeaways on why this approach has been successful for us, focusing on the technology decisions, including feature flags but also calling out other “star players” such as serverless and engineer-owned IAC.

Section 4:  Deep-Dive: Data Migrations for Service-Oriented Architecture

This section will “zoom in” on one of the pieces of our blueprint: Data Migrations.  Data Migrations can be especially challenging for teams considering modernization, and this has been true for the Naviance team as well.  This section offers an opportunity to speak to real challenges our team has faced when attempting to migrate customer data that is, in some cases, decades old. 

One critical takeaway from this section will be the key capability to “look ahead” at the results of a data migration in a production environment.  This is achieved by smart use of feature flags, as well as an architecture that allows data migrations to be “tested in production” with zero impact to customers.  After reviewing a few tactical approaches we take in this area, the conclusion of this section will reflect on the reality that data migrations have no “silver bullet” which makes them easy; smart teams should expect to find surprises and plan from the beginning with contingencies and flexibility in mind.

Section 5: Results

The last section of the presentation will dive into a variety of results and metrics which paint the picture of our Modernization journey’s positive results.  The precise statistics which we can release publically may be subject to change pending internal review, but the following items are tracked today internally to measure our progress:

• DORA metrics (especially deployment cadence) on modernized systems
• LOC deleted from our existing Monolith
• Features shipped to market using our Modernization approach
• Team sentiment surveys & team member quotes: how our engineers have responded to feature-flag-oriented development and working on smaller, context-bound systems