BigQuery Data Export

This topic explains how to create and test a BigQuery destination for Data Export. BigQuery is a cloud-based data processing and analysis platform that lets you work with large sets of data. By exporting your LaunchDarkly flag evaluation and experiment data to the same BigQuery warehouse as your other data, you can build custom reports in BigQuery to answer product behavior questions.

Prerequisites

To configure the BigQuery Data Export integration, you must have the following prerequisites:

• You must have a LaunchDarkly base role of Admin, Owner, or Writer, or a custom role with the following permissions:
  • integrations actions: createIntegration, updateConfiguration, and validateConnection
  • destinations actions: createDestination, updateConfiguration, and updateOn

• You must create a VPC firewall rule that allows the LaunchDarkly data transfer service’s static IP address: 35.192.85.117

Choose an authentication method

LaunchDarkly supports two methods for authenticating your BigQuery Data Export destination to your Google Cloud project:

• Service account impersonation: The LaunchDarkly export service account impersonates a service account in your Google Cloud project. This is the default method. To use it, skip to the Create a Google Cloud service account section.
• Workload identity federation: LaunchDarkly authenticates with an OAuth client from your own identity provider (IdP) and exchanges the credentials for Google Cloud access through the Google Security Token Service (STS). To use it, read Set up workload identity federation.

You select the authentication method when you create the destination. You cannot change it after you save the destination.

If the “Authentication method” option does not appear when you create a destination, workload identity federation is not enabled for your account.

Set up workload identity federation

If you use workload identity federation instead of service account impersonation, LaunchDarkly does not need access to your Google Cloud service account. Instead, LaunchDarkly exchanges credentials from your identity provider for a token that impersonates your destination service account. If you are setting up workload identity federation in Google Cloud for the first time, read Google’s Workload identity federation documentation.

Prepare your identity provider and Google Cloud project

Before you configure the destination in LaunchDarkly, you need:

• An OAuth client in your identity provider for LaunchDarkly to use. Save the client ID, the client secret, and your identity provider’s OAuth token endpoint URL.
• A workload identity pool and provider in your Google Cloud project, configured to trust your identity provider. Save the provider’s full resource name.
• A destination service account that holds the BigQuery dataset and Google Cloud Storage staging permissions. To set this up, follow Create a Google Cloud service account and Create staging bucket, and skip Grant LaunchDarkly access to your Google Cloud service account. That section only applies to service account impersonation.
• The “Workload Identity User” role, roles/iam.workloadIdentityUser, granted to your workload identity principal on the destination service account. This lets the federated identity impersonate the destination service account after the token exchange.

Configure the destination with workload identity federation

To configure the destination:

1. Navigate to the Integrations page in LaunchDarkly and find “BigQuery Data Export.”
2. Click Add integration. The “Create a destination” panel appears.
3. Give the integration a human-readable Name.
4. Select a LaunchDarkly Project and environment to export data from.
5. Enter the GCP project ID.
6. Enter your GCS bucket name.
7. Enter your GCS bucket region.
8. Enter an Export dataset ID to name your dataset. If your dataset does not already exist, LaunchDarkly creates it automatically.
9. Under “Authentication method,” select Workload identity federation.
10. Enter your Identity provider token URL. This is the OAuth2 token endpoint of your identity provider.
11. Enter your Client ID.
12. Enter your Client secret. LaunchDarkly does not display the client secret after you save the destination.
13. Enter the Workload identity provider resource name. This is the full resource name of your workload identity pool provider, for example //iam.googleapis.com/projects/123/locations/global/workloadIdentityPools/pool/providers/provider. You can find it in the Google Cloud console under “Workload Identity Federation.”
14. Enter your Destination service account email.
15. (Optional) Enter Identity provider scopes if your identity provider requires scopes in the token request. Separate multiple scopes with spaces.
16. Click Test connection to ensure your configuration is correct.
17. After reading the Integration Terms and Conditions, check the I have read and agree to the Integration Terms and Conditions checkbox.
18. Click Save destination. The new destination appears in the list of destinations.

Create a Google Cloud service account

To begin, we recommend that you use a Google Cloud service account to provide the LaunchDarkly export service account access to your BigQuery warehouse.

To do this, follow the Google Cloud instructions to create a service account. In the “Permissions” step, grant the account the BigQuery User role.

Save the service account email for use later when you set up the integration in LaunchDarkly.

If you already have a Google Cloud service account, here’s how to add the role:

1. In Google Cloud, navigate to Service Accounts under “IAM & Admin” and open your service account for the LaunchDarkly export service.
2. Click on the Permissions tab.
3. Click Manage access.
4. In the Role field, select “BigQuery User.”
5. Click Save.

Your service account now has permission to act as a BigQuery user.

Grant LaunchDarkly access to your Google Cloud service account

Then, grant your LaunchDarkly export service account access to your Google Cloud service account.

To find your LaunchDarkly export service account email address:

1. In LaunchDarkly, navigate to the Integrations page and find “BigQuery Data Export.”
2. Click Add integration. The “Create a destination” panel appears.
3. Copy the Export service account email address and save it for use in the next step.

Then, in Google Cloud, grant LaunchDarkly’s export service account access to your Google Cloud service account:

1. In Google Cloud, navigate to Service Accounts and select the service account you created or configured in the previous section.
2. Click the Principals with access tab.
3. Click Grant access.
4. In the “Add principals” section, paste your LaunchDarkly export service account email address into the New principals field.
5. In the “Assign roles” section, assign the “Service Account User” role in the role field.
6. Click Add another role.
7. Assign the Service account token creator role in the second Role field.
8. Click Save.

Create staging bucket

Next, create a Google Cloud storage bucket for staging data, and grant your service account access to it.

To do this:

1. Create a Google Cloud storage bucket. Save the bucket name for use later when you set up the integration in LaunchDarkly.
2. In the “Choose where to store your data” section, choose a Region that matches the location of your destination dataset in BigQuery. Save this region for use later when you set up the integration in LaunchDarkly.
3. From the bucket’s Permissions tab, grant your service account access to the bucket
   • In the Select a role field, assign the “Storage Admin” role.

Configure the BigQuery Data Export integration

Finally, in LaunchDarkly, configure the BigQuery Data Export integration.

To do this, you first need to retrieve the following information from your Google Cloud account:

• Your GCP project ID.
• The GCS bucket name and region from the bucket you created in the previous step.
• Your Destination service account email from the service account you created in a previous step. It will be formatted like accountname@abc123.gserviceaccount.com.

Then configure the integration:

1. In LaunchDarkly, return to the Integrations page and find “BigQuery Data Export.”
2. Click Add integration. The “Create a destination” panel appears.
3. Give the integration a human-readable Name.
4. Select a LaunchDarkly Project and environment to export data from.
5. Enter the GCP project ID.
6. Enter your GCS bucket name.
7. Enter your GCS bucket region.
8. Enter an Export dataset ID to name your dataset. If your dataset does not already exist, LaunchDarkly creates it automatically.
9. Enter your Destination service account email.
10. Click Test connection to ensure your configuration is correct.
11. After reading the Integration Terms and Conditions, check the I have read and agree to the Integration Terms and Conditions checkbox.
12. Click Save destination. The new destination appears in the list of destinations.

Your BigQuery Data Export integration is now complete.

To open or review the different event kinds for BigQuery Data Export destinations, read Warehouse Data Export schema reference.