Databricks Data Export

Overview

This topic explains how to create and test a Databricks destination for Data Export. Databricks is a cloud-based data processing and analysis platform that lets you work with large sets of data. By exporting your LaunchDarkly experiment data to the same Databricks warehouse as your other data, you can build custom reports in Databricks to answer product behavior questions.

You can also run experiments using warehouse native metrics. To learn more, read Creating experiments using warehouse native metrics.

Prerequisites

To configure the Databricks Data Export integration, you must have the following prerequisites:

• You must have a LaunchDarkly role that allows you to add and edit integrations and add destinations

• You need to allow LaunchDarkly’s data transfer service’s static IP address: 35.192.85.117

Create a service principal

We recommended that you use a Databricks Service principal to provide LaunchDarkly service access to your warehouse.

To create a new service principal, follow the Databricks instructions to Add service principals to your account.

Ensure that the principal has Workspace access and Databricks SQL access entitlements. To learn how, read the Databricks documentation about Access entitlements.

Create a principal access token

Next, generate and provide an access token for the service principal. You can create an access token using the Databricks REST API. To learn how, read the Databricks REST API documentation.

You must provide specific values to the REST API. In the following example, set these values:

• Replace databricks-instance with your Databricks workspace URL. For example, dbc-abcd1234-5678.cloud.databricks.com.
• Replace your-existing-access-token with an existing valid PAT (string) that has permissions to create new tokens.

Provide the values for these parameters:

$
curl -X POST https://<databricks-instance>/api/2.0/token/create \
>
  -H "Authorization: Bearer <your-existing-access-token>" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "comment": "New PAT using DB API",
>
    "lifetime_seconds": <lifetime-of-pat-in-seconds>
>
  }'

Omitting lifetime_seconds sets the lifetime to the maximum allowed by the workspace configuration.

You will provide the access token later when setting up the integration in LaunchDarkly.

Create a SQL warehouse

Now, follow the Databricks instructions to Create a new SQL warehouse.

Under permissions, assign the Can use permission to the service principal you created above.

Under Connection details, find and save the Server hostname and HTTP path of the warehouse. You will need these when setting up the integration in LaunchDarkly.

Give catalog and schema permissions

Then, give the service principal you created the appropriate permissions for your destination catalog and destination schema.

For your destination catalog, give the service principal the USE CATALOG permission.

For your destination schema, give the service principal the following grants:

• USE SCHEMA
• APPLY TAG
• MODIFY
• READ VOLUME
• SELECT
• WRITE VOLUME
• CREATE MATERIALIZED VIEW
• CREATE TABLE
• CREATE VOLUME

Databricks selects “Unity Catalog” for metastore type by default. If your workspace uses the legacy Hive metastore, select it instead and provide the required S3 bucket details and access keys.

Configure the Databricks Data Export integration

Finally, in LaunchDarkly, configure the Databricks Data Export integration.

To do this:

1. In LaunchDarkly, navigate to the Integrations page and find “Databricks Data Export.”
2. Click Add integration. The “Create a destination” panel appears.
3. Give the integration a human-readable Name.
4. Select a Project and environment to export data from.
5. Enter the Databricks Server hostname you saved when you created a SQL warehouse.
6. Enter the HTTP path of your Databricks server you saved when you created a SQL warehouse.
7. Enter the Databricks catalog.
8. Enter the Databricks Schema name.
9. Enter the principal Access token you created in a previous step.
10. Select a Metastore of “Unity Catalog” or “Hive.”
11. Click Test connection to ensure your configuration is correct.
12. After reading the Integration Terms and Conditions, check the I have read and agree to the Integration Terms and Conditions checkbox.
13. Click Save destination. The new destination appears in the list of destinations.

Your Databricks Data Export integration is now complete.

To view the different event kinds for Databricks Data Export destinations, read Warehouse Data Export schema reference.