Datasets

This topic explains how to create, manage, and use datasets in offline evaluations. Datasets define the inputs AgentControl uses to evaluate model behavior before release. Datasets are only used during offline evaluations.

Offline evaluations run config variations or LLM inputs against datasets that you upload. They score the outputs of those evaluations using criteria such as built-in scorers or judges defined as AgentControl configs. You can reuse the same dataset across multiple evaluation runs. This lets you compare variations and validate changes before you roll them out.

Datasets

A dataset is a file in CSV or JSONL format. Each row within a dataset represents a single evaluation task that LaunchDarkly evaluates independently during a run.

LaunchDarkly generates one model output per row and evaluates it using the criteria you configure. Each row can include the following fields:

• input: The prompt or request sent to the model.
• expected_output: The ideal or target output for the associated input. After you complete an evaluation, you can use this field to compare what you expected against what the model actually returns.
• metadata: Supporting information provided alongside the input, such as retrieved documents or tool responses.
• variables: Named values that populate placeholders in your config prompt templates at runtime.

Here’s an example dataset row:

{ 
    "input": "Is there a way to add a second admin to our account?", 
    "expected_output": "how_to_question",
    "variables": {"customer.tier": "starter", "customer.region": "APAC", "ticket.channel": "chat"},
    "metadata": {"source": "intercom", "priority": "low"} 
}

Upload and download datasets

Upload a file to create a new dataset or to add it to your library for the first time. Download a dataset when you need its current rows outside of LaunchDarkly.

Upload a dataset

To upload a dataset:

1. From the Agents menu, click Library.
2. Click the Datasets tab.
3. Click + New dataset and select Upload dataset. The “Upload dataset” dialog opens.
4. (Optional) Enter a Name for the dataset. If you don’t specify a name, the dataset will use the name of the file you upload.
5. (Optional) Use the autogenerated value in the Key field or enter your own key value.
6. Select your desired dataset file by dropping it into the dialog or using the click to browse link.
7. Click Save dataset.

After you upload a dataset, LaunchDarkly validates and processes the file for use in evaluation runs. This includes validating the file format, detecting the dataset schema, and enforcing row and size limits. LaunchDarkly also computes a dataset hash for deduplication and stores dataset metadata.

After the file is successfully validated, the dataset “Status” column updates to “Ready”, and the dataset becomes available for evaluation runs. If validation fails, you will receive an error.

Download a dataset

To download a dataset:

1. On the Library page, click the Datasets tab.
2. Click the three-dot overflow menu next to the dataset you want to download.
3. Click Download.

Update dataset settings

After you upload a dataset, it appears on the AgentControl Library page. In addition to using it in evaluation runs, you can rename it or edit its description.

Rename a dataset

To rename a dataset:

1. From the Library page, click into the Datasets tab.
2. Find the dataset you want to rename.
3. Click the three-dot overflow menu at the end of the dataset row and click Rename.
4. Enter a new name and click Save.

Manage version history

Every time you save changes, upload a file to append rows, or restore a past version, LaunchDarkly creates a new dataset version. Versions increment in numerical order and record who made the change, when, a short description, and the resulting row count.

Evaluation runs are unaffected by later edits. A run always reads the dataset version that existed at the moment you started it, so editing a dataset later never changes the results of a run you already completed.

View version history

Every dataset keeps a history of its saved versions, so you can review what changed and when.

To view a dataset’s version history:

1. From the Library page, click the Datasets tab.
2. Click the name of the dataset you want to view. This takes you to the dataset Rows tab.
3. Click Version history. A side panel opens listing the dataset’s previous versions.

The version history list shows each version’s number, description, author, timestamp, and row count, starting from the most recent.

To view a past version’s rows, select it from the list. The page shows those rows as read-only and labels which version you’re viewing next to the dataset name.

Restore a previous version

Restoring a version creates a new version with the previous version’s rows. It doesn’t rewind the dataset and it doesn’t undo history.

1. From the dataset’s Rows tab, click Version history.
2. Find the version you want to restore.
3. Click Restore. A “Restore version” dialog opens.
4. Click Restore to confirm restoring the version.

Delete datasets

You can delete a dataset if you no longer need it.

Here’s how:

1. From the Library page, click into the Datasets tab.
2. Click the three-dot overflow menu at the end of the row of the dataset you want to delete.
3. Click Delete. A “Delete dataset” dialog appears.
4. Click Delete dataset.

Datasets in evaluations

Every evaluation run reads from one dataset and pins to the selected dataset version when the run started. Editing the dataset afterward never changes results you already have.

As you use the dataset, the ”# of Evaluations” column in the Datasets tab updates to show how many evaluations have used that dataset.

View evaluations for a dataset

When you configure an offline evaluation, you select a dataset to use as input. To view evaluations that have used a dataset:

1. From the Library page, click the Datasets tab.
2. Click the three-dot overflow menu next to the dataset you want more information about.
3. Choose View evaluations to open the Evaluations page.

This displays all the evaluations based on the dataset.

Dataset rows

Every change you make to a dataset’s rows stages locally in your browser first. Nothing is written to the dataset until you click Save, at which point LaunchDarkly bundles everything you’ve staged into a single new dataset version.

Add rows

Uploading a file works well when you already have many new rows ready to go, for example rows exported from another tool. Adding a row by hand works well for one-off additions, like a single edge case you want to test.

Add rows by upload

You can grow an existing dataset by uploading a file, instead of creating a new dataset each time you have more rows to add.

Here’s how:

1. In the left sidebar, click Agents. The AgentControl menu appears.
2. Click Library.
3. Click into the Datasets tab.
4. Click the name of the dataset you want to view. The dataset details page opens.
5. Click the + Add row menu and select Upload rows. The “Upload rows” dialog opens.
6. Drag and drop your desired file or click the Click to browse link to select a file. The file must be in the same CSV or JSONL format as your original upload.
7. Click Upload rows.

LaunchDarkly parses the file and adds its rows to the end of the dataset without replacing or removing existing rows. If the file fails to parse, LaunchDarkly leaves your existing rows untouched and shows an error.

Each successful upload creates a new dataset version. To learn more, read Manage version history.

Add rows by hand

1. From the Library page, click into the Datasets tab.
2. Click the name of the dataset you want to add rows to. This takes you to the dataset Rows tab.
3. Click the + Add row menu and select Add row. The “Add row” side panel opens.
4. Update the desired fields as needed.
5. Click Add row. Confirming adds a new row to the dataset and creates a new version.

Navigate dataset rows

After you upload a dataset, you can browse and search its rows without leaving the browser.

View dataset rows

To view a dataset’s contents:

1. In the left sidebar, click Agents. The AgentControl menu appears.
2. Click Library.
3. Click the Datasets tab.
4. Click the name of the dataset you want to view. The dataset details page opens.

The rows table shows each row’s input, expected output, variables, and metadata, along with when the row was last updated.

Search dataset rows

Use the search field above the table to find a specific row.

Search matches text in the Input and Expected output fields only. Clear the search field to return to the full row list.

Edit dataset rows

You can edit a dataset’s rows directly from its Rows tab.

Here’s how:

1. From the Library page, click the dataset row and navigate into the Rows tab.
2. Click the row you want to change. The “Edit row” side panel opens.
3. Update the desired fields as needed.
4. Click Add row. Confirming adds a new row to the dataset and creates a new version.

Reorder rows

Row order determines which rows a top-N evaluation run reads first. You can reorder rows by dragging them, or using the row menu.

To move a row to a new position, drag the six-dot handle at the beginning of the row and drop it in the order you want.

To move a row without dragging:

1. Click the three-dot overflow menu at the end of the row.
2. Select Move to top, Move up, Move down, or Move to bottom. The Move to top and Move to bottom apply across the whole dataset, not just the rows currently visible on the page.
3. Click Save to commit everything you’ve moved into a new dataset version.

If you select multiple rows and move them, LaunchDarkly preserves their relative order.

If your save conflicts with someone else’s

If another editor saved changes to the same dataset while you were working, your save fails and LaunchDarkly names the rows that changed. Your other staged changes persist. To resolve this:

1. Refresh the dataset’s rows.
2. Reapply any staged changes that weren’t affected by the conflict.
3. Click Save again.

Delete rows

To delete one or more rows from the dataset:

1. Open the dataset’s details page.
2. Select one or more rows using the checkboxes.
3. Click Delete. Deleted rows are permanently removed from the dataset.