Amazon S3 Data Export

This topic explains how to create and test an Amazon S3 destination for Data Export. Amazon S3 is an object storage service that stores your data as files in a bucket. By exporting your LaunchDarkly flag evaluation and experiment data to an S3 bucket, you can archive the data or load it into your own data pipeline and analytics tools.

Prerequisites

To configure the S3 Data Export integration, you must have the following prerequisites:

• Data Export enabled for the project and environment you want to export from.
• An Amazon S3 bucket, and permissions in AWS to create IAM roles and policies. To learn more, read Creating a bucket in the AWS documentation.
• Access to the AWS Management Console or AWS Command Line Interface (CLI).
• A LaunchDarkly role that allows you to add and edit integrations and add destinations.

How S3 Data Export authenticates

S3 Data Export uses AWS Identity and Access Management (IAM) role-based access. You create an IAM role in your AWS account and grant LaunchDarkly permission to assume that role, so LaunchDarkly can write export files to your bucket without storing long-lived access keys.

LaunchDarkly generates two policies for you during setup:

• An IAM permissions policy that grants write access to your bucket.
• An IAM trust policy that allows the LaunchDarkly data transfer service to assume your role.

You copy both policies from LaunchDarkly into AWS, create the role, and paste the role’s ARN back into LaunchDarkly.

Create an S3 bucket

First, create the bucket that LaunchDarkly writes exported files to. We recommend using a dedicated bucket for LaunchDarkly Data Export to keep the export data separate from other workloads and to scope permissions tightly.

To create a bucket:

1. Navigate to the Amazon S3 console and click Create bucket.
2. Enter a Bucket name and select an AWS Region. Save the bucket and region information to enter into LaunchDarkly in the next section.
3. Adjust the remaining settings as needed. You can keep “Block all public access” enabled.
4. Click Create bucket.

Start the integration in LaunchDarkly

Next, begin configuring the destination in LaunchDarkly:

1. In LaunchDarkly, navigate to the Integrations page and find “S3 Data Export.”
2. Click Add new. The “Create S3 Data Export configuration” page appears.
3. Give the integration a human-readable Name.
4. Select a Project and environment to export data from.
5. Enter the Bucket name of the bucket you created in the previous section.
6. Select the Bucket region that matches the region of your bucket you selected in the previous section.
7. (Optional) Enter a Folder name. This is a prefix within the bucket that LaunchDarkly writes exported files under. If you leave it blank, LaunchDarkly uses ld_export.
8. Select an Exported file format. Parquet is the default and recommended format. To learn more, read Supported file formats.
9. Click Generate policies. LaunchDarkly displays an IAM permissions policy and an IAM trust policy. Copy each policy to your clipboard using its copy button. You use these policies in the following two sections.

Keep this page open, as you return to it in the following sections.

Create the IAM policy in AWS

Use the IAM permissions policy you copied to create a policy in AWS:

1. In the AWS IAM console, navigate to Policies and click Create policy.
2. Select the JSON editor.
3. Paste the IAM permissions policy you copied from LaunchDarkly.
4. Click Next, name the policy, and click Create policy.

You attach this policy to the role in the next section.

Create the IAM role in AWS

Next, create the IAM role that LaunchDarkly assumes:

1. In the AWS IAM console, navigate to Roles and click Create role.
2. Select Custom trust policy.
3. Paste the IAM trust policy you copied from LaunchDarkly, then click Next.
4. Select the permissions policy you created in the previous section, then click Next.
5. Enter a Role name, then click Create role.
6. Open the role you created and copy its ARN.

Test and save the configuration

Finally, return to LaunchDarkly to finish setup:

1. Paste the role’s ARN into the IAM role ARN field.
2. After reading the Integration Terms and Conditions, check the I have read and agree to the Integration Terms and Conditions checkbox.
3. Click Test connection and save.

If the connection test succeeds, LaunchDarkly creates the destination and begins the first data transfer. The initial transfer can take 30 to 60 minutes to complete.

Your S3 Data Export destination is now configured.

Supported file formats

S3 Data Export can write exported data in the following file formats:

• Parquet: this is the default and recommended format for its compression and query performance
• CSV
• JSON
• JSON Lines (JSONL)

How data is organized in your bucket

LaunchDarkly writes exported data as files in Apache Hive-style partitions, using the following path within your bucket:

<folder>/<model_name>/dt=<transfer_date>/<file_part>_<transfer_timestamp>.<extension>

In this path, <folder> is the folder name you configured, and <model_name> is the name of the exported data model. Each transfer can write more than one file per model.

To signal when a transfer completes, LaunchDarkly writes a manifest file for each model under a _manifests folder at the root of your configured folder. Downstream pipelines can watch for these manifest files to determine when a transfer’s data is ready to read.

Use a bucket with KMS encryption

If your bucket is encrypted with an AWS Key Management Service (KMS) customer-managed key (CMK), the IAM permissions policy also needs permission to use that key. Add the following statement to the Statement array of the IAM permissions policy before you create the policy in AWS, and replace REGION, ACCOUNT_ID, and KEY_ID with your values:

1
{
2
  "Effect": "Allow",
3
  "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
4
  "Resource": "arn:aws:kms:REGION:ACCOUNT_ID:key/KEY_ID"
5
}

Buckets encrypted with server-side encryption with customer-provided keys (SSE-C) are not supported.