Databricks Data Export | LaunchDarkly

Data Export is an add-on feature

Data Export is available as an add-on feature to select plans. To learn more, read about our pricing. To add Data Export to your plan, contact Sales.

This topic explains how to create and test a Databricks destination for Data Export. Databricks is a cloud-based data processing and analysis platform that lets you work with large sets of data. By exporting your LaunchDarkly experiment data to the same Databricks warehouse as your other data, you can build custom reports in Databricks to answer product behavior questions.

You can also run experiments using warehouse native metrics. To learn more, read Creating experiments using warehouse native metrics.

To set up Databricks Data Export, you must:

Prerequisites

To configure the Databricks Data Export integration, you must have the following prerequisites:

You must have a LaunchDarkly role that lets you add and edit integrations and add destinations

One Data Export destination per environment

You can configure only one Data Export destination per LaunchDarkly environment, regardless of destination kind. If you need more than one Data Export destination for the same data, configure them in different environments.

Create a service principal

We recommended that you use a Databricks Service principal to provide LaunchDarkly service access to your warehouse.

To create a new service principal, follow the Databricks instructions to Add service principals to your account.

Ensure that the principal has Workspace access and Databricks SQL access entitlements. To learn how, read the Databricks documentation about Access entitlements.

Choose an authentication method

LaunchDarkly supports two methods for authenticating the Databricks Data Export integration:

OAuth M2M: LaunchDarkly authenticates with the client ID and client secret of a Databricks service principal. This is the default method when the authentication method control appears in the configuration form.
Personal access token: LaunchDarkly authenticates with a Databricks personal access token (PAT).

You select the authentication method when you create the integration. You cannot change it after you save the configuration. To use different OAuth credentials or to switch methods, delete the integration and create a new one.

If the “Authentication method” option does not appear in the configuration form, OAuth M2M is not enabled for your account. Use a personal access token instead.

Expand Create OAuth M2M credentials

Create OAuth M2M credentials

LaunchDarkly connects to your warehouse as the service principal using OAuth machine-to-machine (M2M) authentication. Generate an OAuth secret for the service principal, then provide its client ID and secret to LaunchDarkly. To learn more, read the Databricks documentation about OAuth machine-to-machine (M2M) authentication.

To generate the credentials:

In the Databricks account console, click Settings, then open Identity and access and find “Service principals.”
Select the service principal you created, then copy its Application ID. This is your client ID.
Open the service principal’s Secrets tab, then click Generate secret.
Set a Lifetime (days) for the secret, then click Generate.
Copy the Secret and save it in a secure location. Databricks displays the secret only once. This is your client secret.

You will provide the client ID and secret later when setting up the integration in LaunchDarkly. LaunchDarkly uses the client secret to connect to Databricks but does not store it after setup.

Expand Create a personal access token

Create a personal access token

Use this procedure if you select Personal access token when you set up the integration. First create a workspace token in Databricks. Then use that token to create the PAT that LaunchDarkly uses. Finally, grant your service principal permission to use personal access tokens.

To create a workspace token:

Navigate to Workspace settings > Developer.
Click Manage.
Click Generate new token.
Copy and save the token. You use this token as the bearer token in the next request.

Create the PAT that LaunchDarkly uses by sending the following request. Replace DATABRICKS_HOSTNAME with your workspace hostname, EXISTING_TOKEN with the token from the previous steps, and LIFETIME_SECONDS with the token lifetime in seconds:

API access token

$ curl -X POST https://DATABRICKS_HOSTNAME/api/2.0/token/create \
>   -H "Authorization: Bearer EXISTING_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{
>     "comment": "New PAT",
>     "lifetime_seconds": LIFETIME_SECONDS
>   }'

Copy and save the PAT from the response. Provide this PAT to LaunchDarkly when you set up the integration.

Then grant the service principal permission to use personal access tokens:

Navigate to Workspace settings > Workspace admin > Advanced > Personal access tokens > Permission settings.
Add the service principal you created.
Assign the Can use permission.

Create a SQL warehouse

Now, follow the Databricks instructions to Create a new SQL warehouse.

Under permissions, assign the Can use permission to the service principal you created above.

Under Connection details, find and save the Server hostname and HTTP path of the warehouse. You will need these when setting up the integration in LaunchDarkly.

Give catalog and schema permissions

Then, give the service principal you created the appropriate permissions for your destination catalog and destination schema.

For your destination catalog, give the service principal the USE CATALOG permission.

For your destination schema, give the service principal the following grants:

USE SCHEMA
APPLY TAG
MODIFY
READ VOLUME
SELECT
WRITE VOLUME
CREATE MATERIALIZED VIEW
CREATE TABLE
CREATE VOLUME

Databricks selects “Unity Catalog” for metastore type by default. If your workspace uses the legacy Hive metastore, select it instead and provide the required S3 bucket details and access keys.

Set up the integration

To configure the integration in LaunchDarkly:

Click the gear icon in the left sidebar to view Organization settings.
Click Integrations and find “Databricks Data Export.”
Click Add integration. The Databricks Data Export configuration page appears.
Give the integration a human-readable Name.
Choose a Project and environment to export data from. Environments that already have a Data Export destination are unavailable.
Enter the Server hostname you saved when you created a SQL warehouse. You can enter a bare hostname or prefix it with https://, for example dbc-abcd1234-5678.cloud.databricks.com or https://dbc-abcd1234-5678.cloud.databricks.com. Do not use http://. The value must be a hostname, with no path and no extra whitespace.
Enter the HTTP path you saved when you created a SQL warehouse. The path starts with /sql/, for example /sql/1.0/warehouses/abc123.
Enter the Catalog and Schema to export data to. Catalog and schema names can contain only letters, numbers, and underscores. Hyphens are not allowed.
Select an Authentication method. LaunchDarkly selects OAuth M2M by default. Click Personal access token to authenticate with a token instead.
Enter the credentials for the method you selected:
- OAuth M2M: The Client ID and Client secret you created in a previous step. LaunchDarkly uses these credentials to connect to Databricks but does not store the client secret after setup.
- Personal access token: The PAT you created in a previous step.
Select a Metastore of “Unity Catalog” or “Hive.” LaunchDarkly selects “Unity Catalog” by default.
Enter the staging bucket details if you selected the “Hive” metastore: the S3 bucket name, the S3 bucket region, the Bucket access ID, and the Bucket secret key. LaunchDarkly uses the bucket secret key to connect but does not store it after setup.
Read the Integration Terms and Conditions, then select the I have read and agree to the Integration Terms and Conditions checkbox.
Click Test connection and save. LaunchDarkly tests the connection and, if it succeeds, activates the destination and starts the initial transfer.

Your Databricks Data Export destination is now configured.

To learn about the different event kinds for Databricks Data Export destinations, read Warehouse Data Export schema reference.

View an existing destination

The destination setup is visible on the edit page so that you can confirm which host, HTTP path, catalog, schema, metastore, and client ID a destination uses. However, all fields on the edit page are read-only.

LaunchDarkly never stores the secrets you provide at setup, so it cannot display them again. The Client secret field and the Bucket secret key field for a Hive metastore show a masked placeholder to indicate that you provided a secret when you created the destination.

To view an existing Databricks Data Export destination:

Click the gear icon in the left sidebar to view Organization settings.
Click Integrations and find “Databricks Data Export.”
Find the destination you want to view and click Edit. The Databricks Data Export configuration page appears with the destination’s settings.

To verify that the destination still works, click Send event to send a test event from LaunchDarkly to your warehouse. To change any connection value, delete the destination and configure a new one.

Health checks

If your warehouse Data Export integration develops connection problems or other errors, the integration displays a health check warning. To learn more, read Warehouse health checks.