ClickHouse Data Export

This topic explains how to create and test a ClickHouse destination for Data Export. By exporting your LaunchDarkly data to the same ClickHouse instance as the rest of your data, you can build custom reports in ClickHouse to answer product behavior questions.

LaunchDarkly stages exported data in an Amazon S3 bucket before it loads the data into ClickHouse. These setup instructions include configuring an S3 staging bucket and the Amazon Web Services (AWS) Identity and Access Management (IAM) that LaunchDarkly needs.

Prerequisites

To configure the ClickHouse Data Export integration, you must have the following prerequisites:

• A LaunchDarkly role that allows you to add and edit integrations and add destinations.
• A ClickHouse instance that is reachable over the public internet on its native TLS port. The default port for ClickHouse Cloud is 9440.
• IAM permissions in AWS to create policies, roles, and S3 buckets.
• The ability to run SQL statements in your ClickHouse SQL console.
• Your ClickHouse Cloud service role ARN. To find it, in ClickHouse Cloud go to Settings, then Network security, then View service details.

• If your ClickHouse instance restricts inbound connections by IP, you must allow the LaunchDarkly data transfer service’s static IP address: 35.192.85.117.

For best performance, create your S3 staging bucket in the same AWS region as your ClickHouse instance.

ClickHouse naming conventions

When you select a project and environment, LaunchDarkly pre-fills default names for the resources the setup creates.

You can keep these defaults or customize them:

• Database: ld_export. Shared across all environments.
• User: ld_export_user_<projKey>__<envKey>, where <projKey> is your project key and <envKey> is your environment key. Unique to a project and environment.
• S3 staging bucket: ld-export-staging-<projKey>--<envKey>. LaunchDarkly changes the name to lowercase and replaces underscores with hyphens, because S3 bucket names cannot contain underscores. Unique to a project and environment.

Set up the integration

To set up ClickHouse Data Export, enter your connection details and staging bucket, set up AWS access, generate a SQL script to run in ClickHouse, then test the connection and save the destination.

We recommend having browser windows open for LaunchDarkly, AWS, and ClickHouse so that you can easily switch between them.

Connect your LaunchDarkly environment to ClickHouse

To connect your LaunchDarkly environment to ClickHouse:

1. In LaunchDarkly, click the gear icon in the left sidebar to view Organization settings.
2. Click Integrations and find “ClickHouse Data Export.”
3. Click Add new. The ClickHouse Data Export configuration page appears.
4. Choose a Project and environment to export data from. Environments that already have a ClickHouse Data Export destination are unavailable.
5. Enter the Host name of your ClickHouse instance.
6. Enter the native TLS Port of your ClickHouse instance. The default port for ClickHouse Cloud is 9440.
7. Enter your AWS S3 staging bucket name.
8. Enter your S3 staging bucket region.
9. (Optional) Update the Database name or User from their defaults.
10. Enter your ClickHouse Cloud service role ARN.
11. Click Generate IAM permission and trust policies. LaunchDarkly returns an IAM permissions policy and an IAM trust policy. Save these for use in the next section.

Configure AWS

Next, configure your AWS staging bucket and permissions:

1. In AWS, create an S3 staging bucket using the name from step 7 in the previous section, in the region you selected in step 8.
2. Create an IAM policy using the permissions policy you generated in the previous section.
3. Create an IAM role using the trust policy you generated in the previous section, and attach the policy you created to that role.
4. Copy and save the ARN of the new role to use in the next section.

Complete the setup in LaunchDarkly and ClickHouse

Now, return to LaunchDarkly to complete the setup:

1. Paste the ARN of the new role into the IAM role ARN field in LaunchDarkly.
2. (Optional) If your ClickHouse instance restricts inbound connections by IP, turn on Allowlist the LaunchDarkly IP. The generated script then restricts the export user to the LaunchDarkly egress IP.
3. Click Generate script. LaunchDarkly generates a SQL setup script.
4. Copy the script and paste it into a text editor. In the CREATE USER statement, replace <password> with a strong password that uses only alphanumeric characters, dashes, and underscores.
5. Return to ClickHouse and run the script in the ClickHouse SQL console. This creates the database and user, and grants the permissions LaunchDarkly needs to write exported data.
6. Back in LaunchDarkly, enter the same password in the Password field.
7. After reading the Integration Terms and Conditions, select the I have read and agree to the Integration Terms and Conditions box.
8. Click Test connection and save. LaunchDarkly tests the ClickHouse connection and, if it succeeds, creates and activates the destination.

Your ClickHouse Data Export integration is now complete.

ClickHouse Data Export runs an hourly job, on the hour, that publishes raw LaunchDarkly data in your ClickHouse instance. You can expect the new data to be available within 10-15 minutes after the hour.

To view the different event kinds for ClickHouse Data Export destinations, read Warehouse Data Export schema reference.

Edit an existing destination

To edit a ClickHouse Data Export destination after you set it up:

1. In LaunchDarkly, click the gear icon in the left sidebar to view Organization settings.
2. Click Integrations and find “ClickHouse Data Export.”
3. Find the destination you want to edit and click Edit. The ClickHouse Data Export configuration page appears with the destination’s current settings.
4. Update the fields you want to change.
5. Click Save configuration.

The project and environment you chose in the initial configuration are permanent and cannot be changed.

Health checks

If your warehouse Data Export integration develops connection problems or other errors, the integration displays a health check warning. To learn more, read Warehouse health checks.