Redshift Data Export

Overview

This topic explains how to create and test a Redshift destination for Data Export. Redshift is a cloud-based data processing and analysis platform that lets you work with large sets of data. By exporting your LaunchDarkly experiment data to the same Redshift warehouse as your other data, you can build custom reports in Redshift to answer product behavior questions.

Prerequisites

To configure the Redshift Data Export integration, you must have the following prerequisites:

• Access to an existing provisioned Amazon Redshift cluster. The cluster must be provisioned, not serverless.
• IAM permissions in AWS to create roles, policies, and S3 buckets.
• A LaunchDarkly role with integration editing permissions.
• Ability to run SQL commands through Redshift Query Editor v2 or a SQL client.
• You must know your cluster endpoint. You can find this in your Redshift account.

• You need to allow LaunchDarkly’s data transfer service’s static IP address: 35.192.85.117

For best performance, your S3 bucket and your Redshift cluster should be in the same region.

Redshift naming conventions

Redshift automatically converts all unquoted identifiers including database names, schema names, table names, and column names to lowercase.

LaunchDarkly auto-generates these identifiers:

• Database: ld_export.
• Schema: export_{projkey}__{envkey}.
• Username: ld_export_user_{projkey}__{envkey}.
• S3 bucket: ld-export-staging-{projkey}--{envkey}. LaunchDarkly replaces the underscores with hyphens.

Configure the LaunchDarkly integration

To begin, configure the LaunchDarkly integration with your AWS cluster identifier and endpoint. You can find your AWS cluster information in AWS by navigating to Amazon Redshift > Clusters > [your cluster name].

To add your AWS cluster information in LaunchDarkly:

1. In LaunchDarkly, navigate to the Integrations page and find “Redshift Data Export.”
2. Click Add integration. The “Create a destination” panel appears.
3. Give the integration a human-readable Name.
4. Select a Project and environment to export data from.
5. Enter the Redshift cluster Endpoint of the destination.
6. Enter the Cluster identifier.
7. LaunchDarkly generates a SQL script. Copy this script to your clipboard for use in the Redshift Query Editor v2 or SQL client.

Keep the LaunchDarkly integration configuration panel open, as you will return to it later.

Add LaunchDarkly’s static IP address

Now, add LaunchDarkly’s data transfer service’s static IP address to your cluster’s security group settings:

1. Navigate to your Redshift management console.
2. Click Clusters.
3. Select your Redshift cluster.
4. Click the Properties tab.
5. In the “Network and security settings” section, click the linked security group ID in the VPC security group field. The “Security Groups” page opens.
6. Click the Inbound rules tab.
7. Click Edit.
8. Click Add Rule and populate the following fields:
   • Type: Custom TCP
   • Port Range: 5439 or your Redshift port
   • Source: Select “Custom” and enter 35.192.85.117
9. Click Save rules.

LaunchDarkly’s data transfer service can now connect to your Redshift cluster for data export.

Connect to your Redshift cluster and execute the SQL script

Before running the LaunchDarkly-generated SQL script, connect to your Redshift cluster in Query Editor v2:

1. Open Redshift Query Editor v2 in the AWS console.
2. Under “Clusters,” choose the correct cluster.
3. In the “Edit connection” dialog, elect the appropriate authentication method that has permissions to create the database, create a user, and grant privileges.
4. Select the secret associated with your cluster credentials in the Secret menu. This will look something like redshift:redshift-data-export-cluster-1-awsuser.
5. After connecting, confirm you can see your cluster’s databases in the sidebar.
6. In a new query tab, paste the SQL script you copied from LaunchDarkly.
7. Click Run.

Running the SQL script creates the following database objects in Redshift:

• ld_export: The export database.
• ld_export_user_{projkey}__{envkey}: The service user LaunchDarkly uses to connect using IAM auth. LaunchDarkly auto-generates {projkey}__{envkey} using your project and environment keys.
• CREATE and TEMPORARY privileges: Grants LaunchDarkly permission to create tables and load data.

Create an S3 staging bucket in AWS

Next, return to LaunchDarkly and AWS to create an S3 staging bucket. LaunchDarkly uses S3 as a temporary staging area before loading data into Redshift.

To create the staging bucket:

1. Return to the integration configuration panel in LaunchDarkly.
2. Copy the S3 bucket name that appears in the Create S3 bucket on AWS field.
3. Select an S3 staging bucket region. This should be the same region as your Redshift cluster.
4. Navigate to your AWS S3 console and click Create bucket.
5. Paste the Bucket name that you copied in step 2.
6. Ensure the region you select matches the region you entered in step 3.
7. Click Create bucket.

Create IAM policy

Now, create an IAM policy that allows LaunchDarkly to write to your Redshift cluster.

1. Return to the integration configuration panel in LaunchDarkly.
2. Copy the IAM policy that appears in the IAM Policy field.

3. In your AWS console, navigate to Policies.
4. Click Create policy.
5. Select the JSON editor and replace <ACCOUNT-ID> with your AWS account ID.
6. Paste the IAM policy you copied from LaunchDarkly into the policy document field.
7. Add any policy details and create the policy.

You will use the policy you just created in the next section.

Create IAM role

Next, create an IAM role using the policy you just created:

1. In the integration configuration panel in LaunchDarkly, copy the IAM role that appears in the IAM role field.
2. In your AWS console, navigate to Roles.
3. Click Create role.
4. Select a Custom trust policy.
5. Paste the IAM role you copied from LaunchDarkly into the role document field.
6. Select the IAM policy you created in the previous section.
7. Add any role details and create the role.
8. Copy the ARN of the new role.
9. In LaunchDarkly, paste the ARN of the new role into the IAM role ARN field.

Test and save the configuration

Finally, test your LaunchDarkly setup and save your configuration:

1. In the integration configuration panel in LaunchDarkly, click Test connection. If successful, a confirmation message appears.
2. After reading the Integration Terms and Conditions, check the I have read and agree to the Integration Terms and Conditions checkbox.
3. Click Save destination.

Your Redshift Data Export destination is now configured.