Databricks native Experimentation

Overview

This topic explains how to set up the Databricks native Experimentation integration. Databricks native Experimentation enables you to run experiments in LaunchDarkly using data directly from your AWS Databricks data warehouse. This lets LaunchDarkly read and analyze your metrics data stored in Databricks to power your experiment results, without requiring you to send event data through LaunchDarkly’s SDKs.

Before you can begin running Databricks native experiments, you must configure the Databricks Data Export integration in LaunchDarkly.

Prerequisites

Before setting up Databricks native Experimentation, ensure that you have:

• The Databricks Data Export integration set up in LaunchDarkly.
• Administrative access to your Databricks instance with admin permissions to access the following:
  • Databricks workspace host URL
  • Warehouse HTTP path
  • Catalog and schema names
  • Metrics database and schema names
  • Personal access token (PAT) creation
• An active Databricks cluster
• A LaunchDarkly role of Owner or Admin, or a custom role that allows the following actions:
  • add and edit integrations
  • add destinations
  • add metric data sources
• Event data in your Databricks database that you want to use for Experimentation
• An allow list in Databricks that includes the following outbound IP addresses:

• 52.21.152.96/32
• 52.200.35.24/32
• 52.200.50.23/32
• 54.144.218.89/32
• 54.221.221.197/32
• 34.236.6.43/32

For a full list of LaunchDarkly’s public IP addresses, read Public IP list.

Step 1: Gather your Databricks cluster information

To begin, collect the server hostname and HTTP path for your Databricks cluster:

1. In Databricks, navigate to SQL > SQL Warehouses.
2. Select the warehouse you plan to use. This should be the same warehouse you set up for Databricks Data Export.
3. Select the Connection details tab.
4. Copy and save the following for use in the next section:
   • Server hostname
   • HTTP path

Then, collect and save the Databricks catalog name, schema name, metrics database name, and metrics schema name:

• Catalog name: The name of the Unity catalog where you will store LaunchDarkly data
• Schema name: The name of the schema where Databricks will create computed tables
• Metrics database name: The name of the database where Databricks will store metric events
• Metrics schema name: The name of the metrics schema within the Unity catalog

You will use this information in the following sections.

Step 2: In Databricks, create a PAT

To create a Databricks PAT:

1. Navigate to the User tab.
2. Select Workspace settings > Developer.
3. Click Manage.
4. Click Generate new token.
5. Copy and save the PAT for use in the next section.
6. Navigate to Workspace settings > Workspace admin > Advanced Personal access tokens > Permission settings.
7. Enable the PAT for the workspace, and ensure that it has “can use” or “can manage” permissions.

Step 3: Configure the LaunchDarkly integration

Now, you will configure the Databricks native Experimentation integration in LaunchDarkly. This integration looks similar to the Databricks Data Export integration, but is a separate integration requiring a separate setup process.

To set up the integration:

1. Navigate to the Integrations page.
2. Search for and select Databricks Native Experimentation.
3. Click Add integration. The configuration panel appears.

4. Enter an integration Name.
5. Select the LaunchDarkly Project and environment for this integration. You cannot change this after you save the integration.
6. Enter the server hostname you saved from Step 1 into the Databricks workspace host field.
7. Enter the PAT you saved from Step 2 into the Personal Access token field.
8. Enter the HTTP path you saved from Step 1 into the Warehouse HTTP path field.
9. Enter your Databricks Catalog name.
10. Enter your Databricks Schema name.
11. Enter your Databricks Metrics database name.
12. (Optional) Enter your Databricks Metrics schema name.
13. After reading the Integration Terms and Conditions, check the I have read and agree to the Integration Terms and Conditions checkbox.
14. Click Save configuration.

Step 4: Verify the connection

Finally, verify the connection. On the integration configuration page, check the “Connection status” indicator. Successful connections are indicated by a green checkmark and display a “Connected” status.

Next steps

Read the following topics to understand how to create an experiment using a Databricks native metric and how to analyze the experiment results:

• Creating experiments using warehouse native metrics
• Analyzing experiments

The results page for a warehouse native experiment displays the date and time the results were last updated from Databricks. The initial load of experiment results can take up to 60 minutes to appear, and further results updates appear within 15–30 minutes.