Amazon S3 Data Export - LaunchDarkly

Data Export is an add-on featureData Export is available as an add-on feature to select plans. To learn more, read about our pricing. To add Data Export to your plan, contact Sales.

This topic explains how to create and test an Amazon S3 destination for Data Export. Amazon S3 is an object storage service that stores your data as files in a bucket. By exporting your LaunchDarkly flag evaluation and experiment data to an S3 bucket, you can archive the data or load it into your own data pipeline and analytics tools.

Prerequisites

To configure the S3 Data Export integration, you must have the following prerequisites:

Data Export enabled for the project and environment you want to export from.
An Amazon S3 bucket, and permissions in AWS to create IAM roles and policies. To learn more, read Creating a bucket in the AWS documentation.
Access to the AWS Management Console or AWS Command Line Interface (CLI).
A LaunchDarkly role that allows you to add and edit integrations and add destinations.

How S3 Data Export authenticates

S3 Data Export uses AWS Identity and Access Management (IAM) role-based access. You create an IAM role in your AWS account and grant LaunchDarkly permission to assume that role, so LaunchDarkly can write export files to your bucket without storing long-lived access keys. LaunchDarkly generates two policies for you during setup:

An IAM permissions policy that grants write access to your bucket.
An IAM trust policy that allows the LaunchDarkly data transfer service to assume your role.

You copy both policies from LaunchDarkly into AWS, create the role, and paste the role’s ARN back into LaunchDarkly.

Create an S3 bucket

First, create the bucket that LaunchDarkly writes exported files to. We recommend using a dedicated bucket for LaunchDarkly Data Export to keep the export data separate from other workloads and to scope permissions tightly. To create a bucket:

Navigate to the Amazon S3 console and click Create bucket.
Enter a Bucket name and select an AWS Region. Save the bucket and region information to enter into LaunchDarkly in the next section.
Adjust the remaining settings as needed. You can keep “Block all public access” enabled.
Click Create bucket.

Start the integration in LaunchDarkly

Next, begin configuring the destination in LaunchDarkly:

In LaunchDarkly, navigate to the Integrations page and find “S3 Data Export.”
Click Add new. The “Create S3 Data Export configuration” page appears.
Give the integration a human-readable Name.
Select a Project and environment to export data from.
Enter the Bucket name of the bucket you created in the previous section.
Select the Bucket region that matches the region of your bucket you selected in the previous section.
(Optional) Enter a Folder name. This is a prefix within the bucket that LaunchDarkly writes exported files under. If you leave it blank, LaunchDarkly uses ld_export.
Select an Exported file format. Parquet is the default and recommended format. To learn more, read Supported file formats.
Click Generate policies. LaunchDarkly displays an IAM permissions policy and an IAM trust policy. Copy each policy to your clipboard using its copy button. You use these policies in the following two sections.

When you click Generate policies, LaunchDarkly saves a pending destination named S3 Data Export - ENVIRONMENT-KEY, where ENVIRONMENT-KEY is the key of the environment you selected. You can keep this page open while you work in AWS, or close it and finish setup later. To finish setup later:

Find the pending destination on the Integrations page under “S3 Data Export” and open it. The pending destination does not keep the details you entered.
Re-enter the bucket name, bucket region, folder name, and file format, and change the Name if you want a custom one. The file format resets to Parquet. Select CSV, JSON, or JSONL again if you chose one of those.
Click Generate policies again.
Continue with Test and save the configuration.

Each environment can have only one S3 Data Export destination. If an environment already has a completed S3 destination, you must delete it before you can create a new one.

Create the IAM policy in AWS

Use the IAM permissions policy you copied to create a policy in AWS:

In the AWS IAM console, navigate to Policies and click Create policy.
Select the JSON editor.
Paste the IAM permissions policy you copied from LaunchDarkly.
Click Next, name the policy, and click Create policy.

You attach this policy to the role in the next section. The generated IAM permissions policy looks like this, with BUCKET_NAME replaced by your bucket name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::BUCKET_NAME/*"
    }
  ]
}

LaunchDarkly needs s3:PutObject to write exported files. It needs s3:DeleteObject only for the connection test that runs when you first save the destination. You can remove s3:DeleteObject from the policy after setup is complete. If you remove s3:DeleteObject and later create a new destination with the same IAM role, add it back before you test the connection.

Create the IAM role in AWS

Next, create the IAM role that LaunchDarkly assumes:

In the AWS IAM console, navigate to Roles and click Create role.
Select Custom trust policy.
Paste the IAM trust policy you copied from LaunchDarkly, then click Next.
Select the permissions policy you created in the previous section, then click Next.
Enter a Role name, then click Create role.
Open the role you created and copy its ARN.

Test and save the configuration

Finally, return to LaunchDarkly to finish setup:

Paste the role’s ARN into the IAM role ARN field.
After reading the Integration Terms and Conditions, check the I have read and agree to the Integration Terms and Conditions checkbox.
Click Test connection and save.

If the connection test succeeds, LaunchDarkly creates the destination and begins the first data transfer. The initial transfer can take 30 to 60 minutes to complete. Your S3 Data Export destination is now configured.

Editing an existing destination

After setup is complete, you can change only the destination’s Name and Maintainer. You can change the Name using the edit button next to the destination name, and the Maintainer from the side panel. The project and environment, bucket name, bucket region, folder name, exported file format, and IAM role ARN are permanent and you cannot change them. If you need to update these settings, delete the destination and create a new one.

Health checks

If your S3 Data Export integration develops connection problems or other errors, the integration displays a health check warning. To learn more, read Warehouse health checks.

Supported file formats

S3 Data Export can write exported data in the following file formats:

Parquet: this is the default and recommended format for its compression and query performance
CSV
JSON
JSON Lines (JSONL)

How data is organized in your bucket

LaunchDarkly writes exported data as files in Apache Hive-style partitions, using the following path within your bucket:

<folder>/<model_name>/dt=<transfer_date>/<file_part>_<transfer_timestamp>.<extension>

In this path, <folder> is the folder name you configured, and <model_name> is the name of the exported data model. Each transfer can write more than one file per model. To signal when a transfer completes, LaunchDarkly writes a manifest file for each model under a _manifests folder at the root of your configured folder. Downstream pipelines can watch for these manifest files to determine when a transfer’s data is ready to read.

Exported data can contain duplicate recordsObject storage is append-only, and LaunchDarkly re-scans a lookback window on each transfer to make sure no data is missed. As a result, the same record can appear in more than one transfer. Deduplicate on the primary key in your downstream pipeline, keeping the most recent record.

Use a bucket with KMS encryption

If your bucket is encrypted with an AWS Key Management Service (KMS) customer-managed key (CMK), the IAM permissions policy also needs permission to use that key. Add the following statement to the Statement array of the IAM permissions policy before you create the policy in AWS, and replace REGION, ACCOUNT_ID, and KEY_ID with your values:

{
  "Effect": "Allow",
  "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
  "Resource": "arn:aws:kms:REGION:ACCOUNT_ID:key/KEY_ID"
}

Buckets encrypted with server-side encryption with customer-provided keys (SSE-C) are not supported.

​Prerequisites

​How S3 Data Export authenticates

​Create an S3 bucket

​Start the integration in LaunchDarkly

​Create the IAM policy in AWS

​Create the IAM role in AWS

​Test and save the configuration

​Editing an existing destination

​Health checks

​Supported file formats

​How data is organized in your bucket

​Use a bucket with KMS encryption

Prerequisites

How S3 Data Export authenticates

Create an S3 bucket

Start the integration in LaunchDarkly

Create the IAM policy in AWS

Create the IAM role in AWS

Test and save the configuration

Editing an existing destination

Health checks

Supported file formats

How data is organized in your bucket

Use a bucket with KMS encryption