IntegrationsData ExportWarehouse Data Export

Databricks Data Export

This integration is not available in LaunchDarkly's European Union (EU) instance
To learn more, read LaunchDarkly in the European Union (EU).
Data Export is an add-on feature

Data Export is available as an add-on to select plans. To learn more, read about our pricing. To add Data Export to your plan, contact Sales.

Overview

This topic explains how to create and test a Databricks destination for Data Export. Databricks is a cloud-based data processing and analysis platform that lets you work with large sets of data. By exporting your LaunchDarkly experiment data to the same Databricks warehouse as your other data, you can build custom reports in Databricks to answer product behavior questions.

Create a service principal

We recommended that you use a Databricks Service principal to provide LaunchDarkly service access to your warehouse.

To create a new service principal, follow the Databricks instructions to Add service principals to your account. You can also read LaunchDarkly’s instructions for creating a new service principal.

Ensure that the principal has Workspace access and Databricks SQL access entitlements. To learn how, read the Databricks documentation about Access entitlements.

Create a principal access token

Next, generate and provide an access token for the service principal. You can create an access token using the Databricks REST API. To learn how, read the Databricks REST API documentation.

You must provide specific values to the REST API. In the following example, set these values:

  • Replace databricks-instance with your Databricks workspace URL. For example, dbc-abcd1234-5678.cloud.databricks.com.
  • Replace your-existing-access-token with an existing valid PAT (string) that has permissions to create new tokens.

Provide the values for these parameters:

API access token
$curl -X POST https://<databricks-instance>/api/2.0/token/create \
>  -H "Authorization: Bearer <your-existing-access-token>" \
>  -H "Content-Type: application/json" \
>  -d '{
>    "comment": "New PAT using DB API",
>    "lifetime_seconds": <lifetime-of-pat-in-seconds>
>  }'

Omitting lifetime_seconds sets the lifetime to the maximum allowed by the workspace configuration.

You will provide the access token later when setting up the integration in LaunchDarkly.

Create a SQL warehouse

Now, follow the Databricks instructions to Create a new SQL warehouse.

Under permissions, assign the Can use permission to the service principal you created above.

Under Connection details, find and save the Server hostname and HTTP path of the warehouse. You will need these when setting up the integration in LaunchDarkly.

Give catalog and schema permissions

Then, give the service principal you created the appropriate permissions for your destination catalog and destination schema.

For your destination catalog, give the service principal the USE CATALOG permission.

For your destination schema, give the service principal the following grants:

  • USE SCHEMA
  • APPLY TAG
  • MODIFY
  • READ VOLUME
  • SELECT
  • WRITE VOLUME
  • CREATE MATERIALIZED VIEW
  • CREATE TABLE
  • CREATE VOLUME

Databricks selects “Unity Catalog” for metastore type by default. If your workspace uses the legacy Hive metastore, select it instead and provide the required S3 bucket details and access keys.

Configure the Databricks Data Export integration

Finally, in LaunchDarkly, configure the Databricks Data Export integration.

To do this:

  1. In LaunchDarkly, navigate to the Integrations page and find “Databricks Data Export.”
  2. Click Add integration. The “Create a destination” panel appears.
  3. Give the integration a human-readable Name.
  4. Select a Project and environment to export data from.
  5. Enter the Databricks Server hostname you saved when you created a SQL warehouse.
  6. Enter the HTTP path of your Databricks server you saved when you created a SQL warehouse.
  7. Enter the Databricks catalog.
  8. Enter the Databricks Schema name.
  9. Enter the principal Access token you created in a previous step.
  10. Select a Metastore of “Unity Catalog” or “Hive.”
  11. Click Test connection to ensure your configuration is correct.
  12. After reading the Integration Terms and Conditions, check the I have read and agree to the Integration Terms and Conditions checkbox.
  13. Click Save destination. The new destination appears in the list of destinations.

Your Databricks Data Export integration is now complete.

To view the different event kinds for Databricks Data Export destinations, read Warehouse Data Export schema reference.