BigQuery Data Export

Overview

This topic explains how to create and test a BigQuery destination for Data Export. BigQuery is a cloud-based data processing and analysis platform that lets you work with large sets of data. By exporting your LaunchDarkly flag evaluation and experiment data to the same BigQuery warehouse as your other data, you can build custom reports in BigQuery to answer product behavior questions.

Prerequisites

To configure the BigQuery Data Export integration, you must have the following prerequisites:

• You must have a LaunchDarkly base role of Admin, Owner, or Writer, or a custom role with the following permissions:
  • integrations actions: createIntegration, updateConfiguration, and validateConnection
  • destinations actions: createDestination, updateConfiguration, and updateOn

Create a Google Cloud service account

To begin, we recommend that you use a Google Cloud service account to provide LaunchDarkly’s export service account access to your BigQuery warehouse.

To do this, follow the Google Cloud instructions to create a service account. In the “Permissions” step, grant the account the BigQuery User role.

Save the service account email for use later when you set up the integration in LaunchDarkly.

If you already have a Google Cloud service account, here’s how to add the role:

1. In Google Cloud, navigate to Service Accounts under “IAM & Admin” and open your service account for the LaunchDarkly export service.
2. Click on the Permissions tab.
3. Click Manage access.
4. In the Role field, select “BigQuery User.”
5. Click Save.

Your service account now has permission to act as a BigQuery user.

Grant LaunchDarkly access to your Google Cloud service account

Then, grant your LaunchDarkly export service account access to your Google Cloud service account.

To find your LaunchDarkly export service account email address:

1. In LaunchDarkly, navigate to the Integrations page and find “BigQuery Data Export.”
2. Click Add integration. The “Create a destination” panel appears.
3. Copy the Export service account email address and save it for use in the next step.

Then, in Google Cloud, grant LaunchDarkly’s export service account access to your Google Cloud service account:

1. In Google Cloud, navigate to Service Accounts and select the service account you created or configured in the previous section.
2. Click the Principals with access tab.
3. Click Grant access.
4. In the “Add principals” section, paste your LaunchDarkly export service account email address into the New principals field.
5. In the “Assign roles” section, assign the “Service Account User” role in the role field.
6. Click Add another role.
7. Assign the Service account token creator role in the second Role field.
8. Click Save.

Create staging bucket

Next, create a Google Cloud storage bucket for staging data, and grant your service account access to it.

To do this:

1. Create a Google Cloud storage bucket. Save the bucket name for use later when you set up the integration in LaunchDarkly.
2. In the “Choose where to store your data” section, choose a Region that matches the location of your destination dataset in BigQuery. Save this region for use later when you set up the integration in LaunchDarkly.
3. From the bucket’s Permissions tab, grant your service account access to the bucket
   • In the Select a role field, assign the “Storage Admin” role.

Configure the BigQuery Data Export integration

Finally, in LaunchDarkly, configure the BigQuery Data Export integration.

To do this, you first need to retrieve the following information from your Google Cloud account:

• Your BigQuery project ID.
• The GCS bucket name and region from the bucket you created in the previous step.
• Your Destination service account email from the service account you created in a previous step. It will be formatted like accountname@abc123.gserviceaccount.com.

Then configure the integration:

1. In LaunchDarkly, return to the Integrations page and find “BigQuery Data Export.”
2. Click Add integration. The “Create a destination” panel appears.
3. Give the integration a human-readable Name.
4. Select a LaunchDarkly Project and environment to export data from.
5. Enter the BigQuery Project ID.
6. Enter your GCS bucket name.
7. Enter your GCS bucket region.
8. Enter your Destination service account email.
9. Enter a Schema name for your dataset. If your dataset does not already exist, it will be created automatically.
10. Click Test connection to ensure your configuration is correct.
11. After reading the Integration Terms and Conditions, check the I have read and agree to the Integration Terms and Conditions checkbox.
12. Click Save destination. The new destination appears in the list of destinations.

Your BigQuery Data Export integration is now complete.

To view the different event kinds for BigQuery Data Export destinations, read Warehouse Data Export schema reference.