How to Migrate a Database

In this post, you'll gain a high-level view of database migrations, common terminology, motivation for migrating databases, and common pitfalls along with their solutions. Additionally, you'll get strategic steps to accelerate your migration and increase your confidence and that of your leadership and your stakeholders.

What's the big deal?

Database migrations are risky because, in many cases, your data is your business. Whether you're a bank, an insurance company, social media platform, a manufacturer, or a retailer, without your data, your business could cease to exist. Migrating that data, ensuring it arrives intact, and keeping your mission critical systems whole and running is of the utmost importance.

Not only can a database migration be risky, but it can be challenging as well. It may involve rearchitecting your infrastructure or data, and rethinking access strategy, all while addressing the complexity of data migration and data consistency.

What is a database migration?

Database migration refers to the process of moving data from one database to another and changing applications to consume data from and write data to the newly-migrated database.

As we dig into the details and best practices of database migrations, let's start with some common terminology which will set the stage for the rest of this article.

Homogeneous database migration

When an application and its data are moved from one database to another and the underlying database management systems (DBMS) remains the same.

Examples of homogeneous database migrations

Migrating from SQL Server on-premise to SQL Server hosted on Azure
Migrating from PostgreSQL hosted in EC2 to PostgreSQL on Amazon RDS

Common cases for homogenous database migration

Most commonly, homogeneous database migrations are driven by cost optimization, cluster consolidation, or organizational initiatives like migrating out of data centers and into the cloud.

Heterogeneous database migration

When an application and its data are moved from one database to another and the underlying DBMS engine changes.

Examples of heterogeneous database migrations

Migrating from a relational database management system (RDBMS) like Microsoft SQL Server to PostgreSQL (RDBMS to RDBMS)
Migrating from PostgreSQL to DynamoDB (RDBMS to Serverless Key/Value DBMS)

Common cases for heterogeneous database migration

Most commonly, heterogeneous database migrations are driven by the desire to gain improved capabilities, scale, performance, or cost optimization, and are often found in the move to more cloud-native database implementations.

Application layers

For the purposes of this article, let's assume there are three main layers for an application, for example a web service.

Request/response handling
This layer is responsible for validation of incoming requests, handling the protocol for connections, and providing an interface for interacting with the application logic. Most of all, if well-designed, this will keep the details of protocol and the API model away from your application logic.
Application logic
The "brains" of the application. The rules for handling requests, your calculations, algorithms, and decisions your application makes live here. Application logic should be agnostic of the protocol used for requests and agnostic of any database implementation details.
Data access
The "middle man" between your application logic and your database speaks the language of your database, has retry strategies, and transforms the data from the database into the language of your application, also known as serialization. If well-designed, it conceals the details of your database implementation, including the schema and the underlying database management system.

Now that we've set the stage with some common terminology and understanding, let's jump into some best practices to eliminate the risk from and accelerate your database migration.

Best practices

There are a few essential best practices to contain and minimize the risks while maximizing your ability to react to unanticipated problems with a database migration. Acceleration in database migration has everything to do with confidence. The higher the confidence of your team, your leadership, and your stakeholders, the faster you can go.

Think small

With every change to software, we introduce risk, so the least risky software change is the smallest. The best way to keep software changes small is all about isolation and specialization. Fortunately, the software design principles Separation of Concerns and Single Responsibility Principle offer guidance on how to achieve this in changing an application for a database migration.

Separation of Concerns

This principle asserts that well-designed software should be separated based on the kinds of work it performs.

For each portion of the application which must change, the risk rises considerably as the permutations to be tested grow. When we change our data access layer, other parts of our application should not change, such as the request/response layer or the application logic.

Single Responsibility Principle

Single Responsibility focuses on grouping things together. Some approach Single Responsibility Principle in a reductive way: "each thing should only have one job," but for the purposes of database migration, the latter half of the principle is most relevant:

"objects should have only one responsibility and they should have only one reason to change"

The parts of your code that know how to speak the language of your database, understand how data is stored and structured, and know how to convert data to the model your application uses, should all live close to each other because they will change together as a product of your migration.

Don't redesign your app from the ground up, but do take the time to check your design and make sure you've both separated concerns to reduce the amount of change and ensure that your code that does change will be grouped together so those parts can be highly cohesive.

Decouple release from deployment

Given the risk of a database migration and the expectation that you will have some trial and error, being able to freely deploy, but strategically time and control your release independently of deployment is essential. Feature flags are the best way to decouple release from deployment because they give you the ability to change your application without having to redeploy.

Then

In the past, we turned to configuration files and environment variables, which required recycling an application to pick up changes. It was an effective, albeit clunky, way to change an application without redeploying. Fortunately, you don't have to settle for that approach anymore.

Now

By using a feature management platform, like LaunchDarkly, to manage your feature flags, with a global Flag Delivery Network, you can get your feature flag changes to your application in milliseconds instead of minutes.

Feature flag the data access layer

Use feature flags strategically in your application to gate the use of your newly migrated database.

The toggle

The simplest implementation to gate the use of your new database is the classic toggle. Feature flags are so much more than just on/off switches, but in this particular case, we're going to do just that, create an on/off switch using a straightforward if/then statement in our data access layer.

If our feature flag, use-new-database, is true, our data module will use DynamoDB to fetch the user data, otherwise, it will use Postgres to fetch the user data.

// src/data.ts

import * as ld from 'launchdarkly-node-server-sdk';
import dynamo from './dynamo'
import postgres from './postgres'
import User from './User'

type FindUserParams = {
  id: string;
  currentUser: {
    id: string;
    name: string;
  }
}

async function findUserById({id, currentUser}: FindUserParams): Promise<User> {
  const context = {
    "kind": 'user',
    "key": currentUser.id,
    "name": currentUser.name,
  };

	const useNewDatabase = await client.variation('use-new-database', context, false);

    if (useNewDatabase) {
  	return dynamo.findUserById(userId);
	} else {
	  return postgres.findUserById(userId);
  }
}

(In the above example, the variable useNewDatabase is retrieved using the LaunchDarkly SDK from the feature flag use-new-database and uses the context of the request to target to determine to which database the request should be directed.)

Beyond the boolean

If our database migration is homogenous and we're only changing the location of our new database, then we can completely avoid our if statement and any technical debt we'll have to clean it up later. And because this database configuration is delivered by LaunchDarkly's Flag Delivery Network, we change the connection details of our feature flag easily and serve multiple variations.

// src/data.ts

import * as ld from 'launchdarkly-node-server-sdk';
import postgres from './postgres'
import User from './User'

const databaseCredentials = process.env.DB_CREDENTIALS;

type FindUserParams = {
  id: string;
  currentUser: {
    id: string;
    name: string;
  }
}

const client = ld.init(process.env.LAUNCHDARKLY_SDK_KEY);

async function findUserById({id, currentUser}: FindUserParams): Promise<User> {
  const context = {
    "kind": 'user',
    "key": currentUser.id,
    "name": currentUser.name,
  };

  const {
    hostName,
    port,
    databaseName
  } = await client.variation('database-config', context, false);

  const {
    userId,
    password
  } = databaseCredentials;

 const connectionString = `postgresql://${userId}:${password}/${hostName}:${port}/${databaseName}`;
​
  return postgres.findUserById({
    userId,
    connectionString
  })
​
}
​
export {
 findUserById,
 findUsers,
}
```

(In the above example, the variables hostName, port, databaseName are retrieved using the LaunchDarkly SDK from the feature flag database-config and uses the context of the request to target to determine to which connection string to use when connecting to the database.)

Now that you have three best practices to guide your database migration: think small, decouple release from deployment, and feature flag your data access layer with either a toggle (boolean) or a multivariate (numeric, string, or JSON) flag, let's explore how you can use those to give you comprehensive control over the release of your database migration.

Release strategies

The release strategy for your database migration is the capstone of your migration and must account for your tolerance of downtime, your requirements for data consistency, and the criticality of your system to you and your consumers.

Once you've decoupled release from deployment with LaunchDarkly feature flags, you've got three great strategies that you can use on their own or in any combination that works for your needs.

Kill switch

While the kill switch is used as a toggle, you aren't limited to boolean values with a kill switch. Multivariate feature flags can be turned completely off via a kill switch. In the event of a connectivity, permissions, schematic, serialization issue, or any other blocking issue, you can instantly revert to the old database until the issue is resolved.

Progressive release

LaunchDarkly's targeting allows you to progressively ramping up/down traffic to your new database. You have the power to carefully test the scale of your new database, prevent overloading it and degrading your application performance with the ability to instantly adjust the flow of traffic as you adjust resources to keep up with the load. LaunchDarkly's workflow automation allows you to automate the progression of your release to match your timeline.

Strategic release

When you use LaunchDarkly targeting, you can target any context and direct traffic to the new or old database accordingly. Whether you're beta-testing the database migration with a subset of users, rolling out to your stakeholders first for validation, or targeting the context based on geography, device, or identity, targeting allows you to define where users requests for data are directed at any grain.

Conclusion

In this post, we've laid out some of the fundamental terminology around database migrations, set the stage for how to ensure your application is ready for a smooth migration, decoupling your deployment from your release so you can decide when your features release and have confidence to deploy any time, and given you three time-tested release strategies guaranteed to put you in the driver's seat of your database migration release strategy.

Continue your learning

If you're interested in diving in deeper to the technical implementation of a database migration, be sure to check out Migrating Databases in Kubernetes with LaunchDarkly, on our YouTube channel, where the fabulous Peter McCarron dives right in and shows you hands-on how to migrate databases with LaunchDarkly.

Are you working through an ongoing cloud migration or just planning a cloud migration? Find out how using feature flags can mitigate risk in infrastructure migrations.

Still believe that feature flags are nothing but toggles? Brian Rinaldi takes you beyond the boolean in this post.

There are so many ways to learn how to feature flag and one of the best by far is LaunchDarkly Academy with guided onboarding, live missions, and interstellar certifications!