> ## Documentation Index
> Fetch the complete documentation index at: https://launchdarkly.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Already using Datadog? Here's how to gate and observe your AI agent skills with LaunchDarkly (without OpenTelemetry)

export const BlogAuthor = ({date, avatar, alt, name}) => {
  return <div className="-mb-4">
      <p className="m-0 text-base font-semibold">
        <em>Published {date}</em>
      </p>
      <div className="flex items-center -mt-4">
        {avatar && <img src={avatar} alt={alt} className="h-8 w-8 rounded-full m-0" />}
        <p className="ml-[5px] text-lg">{name}</p>
      </div>
    </div>;
};

<View title="Developer" />

<View title="Federal docs" />

<View title="EU docs" />

<BlogAuthor date="September 23, 2026" avatar="/images/authors/alexis-roberson.png" alt="portrait of Alexis Roberson." name="by Alexis Roberson" />

<Frame caption="Sending Datadog trace data to LaunchDarkly.">
  <img src="https://mintcdn.com/launchdarkly/AAfzx725JxtBqiAm/images/tutorials/datadog-to-launchdarkly-traces-part-one/datadog_launchdarkly_part1_blogpost_photo.png?fit=max&auto=format&n=AAfzx725JxtBqiAm&q=85&s=196edf2f8ffb6a0121a2324d9b8f5fa2" alt="Sending Datadog trace data to LaunchDarkly." width="1280" height="720" data-path="images/tutorials/datadog-to-launchdarkly-traces-part-one/datadog_launchdarkly_part1_blogpost_photo.png" />
</Frame>

If you're already using Datadog to monitor your application, you can [forward your existing application performance monitoring (APM) trace data to LaunchDarkly using the Datadog Agent](/docs/home/observability/datadog-agent). No additional instrumentation required. This unlocks feature flags, Guarded Rollouts, and automatic rollbacks that use Observability data from your existing Datadog configuration.

This tutorial walks through that setup with **Serenia**, an AI assistant for an event space sales team. The higher-risk `qualify_lead` skill is gated by a feature flag. By the end, Datadog APM traces flow into both Datadog and LaunchDarkly.

All code for this tutorial is in the [`tutorial-datadog-part-one` branch](https://github.com/arober39/serenia-agent-skills/tree/tutorial-datadog-part-one) of the Serenia repo.

## Prerequisites

To complete this tutorial, you need:

* A [LaunchDarkly account](https://app.launchdarkly.com/signup) with Writer permissions
* A [Datadog account](https://www.datadoghq.com/) with the Datadog Agent installed
* An [Anthropic API key](https://console.anthropic.com/)
* Cursor, with the [LaunchDarkly Model Context Protocol (MCP) server](https://mcp.launchdarkly.com/mcp/launchdarkly) configured
* Python 3 and Node.js, to run the sample app
* An optional [Airtable account](https://airtable.com/), if you want skills to write lead records

## Serenia: Sales team AI assistant

When a customer sends a message about reserving an event space, the agent classifies their intent and routes it to a skill. Skills can write lead information to Airtable for sales follow-up.

<Frame caption="The Serenia AI assistant UI.">
  <img src="https://mintcdn.com/launchdarkly/AAfzx725JxtBqiAm/images/tutorials/datadog-to-launchdarkly-traces-part-one/serenia-ui-page.png?fit=max&auto=format&n=AAfzx725JxtBqiAm&q=85&s=8989c4ef49ee86f20639fd04f6db13c7" alt="The Serenia AI assistant UI." width="4112" height="2508" data-path="images/tutorials/datadog-to-launchdarkly-traces-part-one/serenia-ui-page.png" />
</Frame>

<Frame caption="Serenia agent architecture.">
  <img src="https://mintcdn.com/launchdarkly/AAfzx725JxtBqiAm/images/tutorials/datadog-to-launchdarkly-traces-part-one/agent_architecture.png?fit=max&auto=format&n=AAfzx725JxtBqiAm&q=85&s=ee729841a9df4f72efe1a44876266d04" alt="Serenia agent architecture." width="1152" height="864" data-path="images/tutorials/datadog-to-launchdarkly-traces-part-one/agent_architecture.png" />
</Frame>

The agent uses these skills:

| Skill | Status | What it does |
| - | - | - |
| `answer_faq` | Stable | Answers venue questions using a knowledge base and Claude |
| `log_inquiry` | Stable | Records prospect contact info to Airtable |
| `qualify_lead` | Flag-gated | Scores leads as hot, warm, or cold and chooses a follow-up action |
| `auto_propose` | Locked | Generates custom event proposals (not used in this tutorial) |

`qualify_lead` is heavier and higher-risk than `log_inquiry`. The `qualify-lead-skill` flag gates it. If the flag is off, the agent falls back to `log_inquiry`.

## Clone the repo

To clone the demo and install dependencies:

<CodeGroup>
  ```shell title="Clone and set up the project" lines wrap theme={null}
  git clone https://github.com/arober39/serenia-agent-skills.git
  cd serenia-agent-skills
  git checkout tutorial-datadog-part-one
  python -m venv venv && source venv/bin/activate
  pip install -r requirements.txt
  ```
</CodeGroup>

## Set up credentials

Copy the example environment file and fill in your keys:

<CodeGroup>
  ```shell title="Copy environment file" lines wrap theme={null}
  cp .env.example .env
  ```
</CodeGroup>

You need credentials from these services:

| Service | Variable(s) | Where to find it |
| - | - | - |
| **Anthropic** | `ANTHROPIC_API_KEY` | [console.anthropic.com](https://console.anthropic.com/) > API Keys |
| **LaunchDarkly** | `LD_SDK_KEY` (server-side SDK key), `LD_CLIENT_SIDE_ID` | Project settings > Environments |
| **Datadog** | `DD_API_KEY`, `DD_SITE` | Organization Settings > API Keys |
| **Airtable** | `AIRTABLE_PAT`, `AIRTABLE_BASE_ID` | [airtable.com/create/tokens](https://airtable.com/create/tokens) |

For `ddtrace`, set `DD_TAGS` to your LaunchDarkly client-side ID. LaunchDarkly uses that tag to route traces to the correct project. Do not use `OTEL_RESOURCE_ATTRIBUTES` for this.

Your `.env` should look like this:

<CodeGroup>
  ```shell title="Example .env file" lines wrap theme={null}
  ANTHROPIC_API_KEY=sk-ant-your-key
  LD_SDK_KEY=sdk-your-key
  LD_CLIENT_SIDE_ID=your-client-side-id
  DD_API_KEY=your-datadog-api-key
  DD_SITE=datadoghq.com
  DD_AGENT_HOST=localhost
  DD_SERVICE=serenia-agent
  DD_TAGS=launchdarkly.project_id:YOUR_CLIENT_SIDE_ID
  AIRTABLE_PAT=patYourToken
  AIRTABLE_BASE_ID=appYourBaseId
  ```
</CodeGroup>

### Configure Airtable

Airtable is optional. If credentials are not set, skills still run and return responses. They skip the customer relationship management (CRM) write.

The `log_inquiry` and `qualify_lead` skills write to a **Leads** table. Create these fields:

| Field | Description |
| - | - |
| Name | Customer's name |
| Email | Customer's email address |
| Message | The customer's original message |
| Status | `"New"` or `"Qualified"` |
| Lead Score | `"Hot"`, `"Warm"`, or `"Cold"` |
| Lead Action | `"Scheduled call"`, `"Sent brochure"`, or `"Sent product info"` |
| Qualification Reason | LLM-generated explanation of the score |

The base ID is in your Airtable URL: `airtable.com/<BASE_ID>/tbl`. Generate a personal access token with `data.records:write` on your base.

To create the base with the Airtable AI assistant, use this prompt:

<Accordion title="Airtable AI assistant prompt">
  ```lines wrap theme={null}
  Create a base called "Serenia CRM" with a single table called "Leads" that has
    the following fields:

   - Name (Single line text)
   - Email (Email)
   - Message (Long text)
   - Status (Single select with options: New, Qualified)
   - Lead Score (Single select with options: Hot, Warm, Cold)
   - Lead Action (Single line text)
   - Qualification Reason (Long text)

   This table will be used to track customer inquiries and qualified leads for an
    event venue. The "log_inquiry" skill creates records with Status "New". The
   "qualify_lead" skill creates records with Status "Qualified" and fills in Lead
    Score, Lead Action, and Qualification Reason.
  ```
</Accordion>

<Frame caption="Using the Airtable AI assistant to create the Serenia CRM base.">
  <img src="https://mintcdn.com/launchdarkly/AAfzx725JxtBqiAm/images/tutorials/datadog-to-launchdarkly-traces-part-one/airtable-chat-assisstant-omni.png?fit=max&auto=format&n=AAfzx725JxtBqiAm&q=85&s=8a1efa780ce66a45b19460f5b4bd6af9" alt="Using the Airtable AI assistant to create the Serenia CRM base." width="996" height="963" data-path="images/tutorials/datadog-to-launchdarkly-traces-part-one/airtable-chat-assisstant-omni.png" />
</Frame>

## Run the app and verify Datadog traces

The sample app is already instrumented with [`ddtrace`](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/python/). Start it, send a request, and confirm traces appear in Datadog APM before you forward anything to LaunchDarkly.

Start the API:

<CodeGroup>
  ```shell title="Start the Serenia API" lines wrap theme={null}
  uvicorn server:app --reload --port 8000
  ```
</CodeGroup>

In a second terminal, start the UI:

<CodeGroup>
  ```shell title="Start the Serenia UI" lines wrap theme={null}
  cd ui
  npm install
  npm run dev
  ```
</CodeGroup>

Generate traces by chatting with the agent at `http://localhost:3000`, or with `curl`:

<CodeGroup>
  ```shell title="Send a test message via curl" lines wrap theme={null}
  curl -X POST http://localhost:8000/api/chat \
    -H "Content-Type: application/json" \
    -d '{"message": "Do you host baby showers?"}'
  ```
</CodeGroup>

Open **Datadog > APM > Traces** and filter by `service:serenia-agent`. You should see traces for each request, with Anthropic calls nested inside skill spans such as `qualify_lead` or `agent.handle_message`.

<Frame caption="Datadog APM traces for service:serenia-agent, with Anthropic calls nested under skill spans.">
  <img src="https://mintcdn.com/launchdarkly/AAfzx725JxtBqiAm/images/tutorials/datadog-to-launchdarkly-traces-part-one/view-dd-traces-view.png?fit=max&auto=format&n=AAfzx725JxtBqiAm&q=85&s=ad64ea7217e1d5808ddeaa20f7aa1af1" alt="Datadog APM traces for service:serenia-agent, with Anthropic calls nested under skill spans." width="3448" height="1920" data-path="images/tutorials/datadog-to-launchdarkly-traces-part-one/view-dd-traces-view.png" />
</Frame>

Those spans come from instrumentation already in the repo:

* **Auto-instrumentation:** `init_tracing()` calls `patch(anthropic=True)` once at startup. Every Anthropic SDK call is traced without wrappers in skill code. Read [`serenia/observability/tracing.py`](https://github.com/arober39/serenia-agent-skills/blob/tutorial-datadog-part-one/serenia/observability/tracing.py).
* **Skill spans:** `trace_skill()` creates a parent span that shows which skill ran, plus tags such as lead score and Airtable record ID. Read how [`qualify_lead.py`](https://github.com/arober39/serenia-agent-skills/blob/tutorial-datadog-part-one/serenia/skills/qualify_lead.py) uses it.

If Airtable credentials are set, new rows appear in Leads:

<Frame caption="Airtable Leads table showing records created by log_inquiry and qualify_lead.">
  <img src="https://mintcdn.com/launchdarkly/AAfzx725JxtBqiAm/images/tutorials/datadog-to-launchdarkly-traces-part-one/serenia-crm-airtable-tracking.png?fit=max&auto=format&n=AAfzx725JxtBqiAm&q=85&s=0ee4fea355c09d55f9700a659ca00c29" alt="Airtable Leads table showing records created by both log_inquiry (Status: New) and qualify_lead (Status: Qualified, with Lead Score and Lead Action populated)." width="1920" height="1002" data-path="images/tutorials/datadog-to-launchdarkly-traces-part-one/serenia-crm-airtable-tracking.png" />
</Frame>

## Create the qualify-lead feature flag

The app already checks a LaunchDarkly flag before `qualify_lead` runs. Create the flag. Evaluations then appear on Datadog spans when the skill runs.

In Cursor, use the LaunchDarkly MCP server with this prompt:

<CodeGroup>
  ```title="Cursor prompt to create the feature flag" lines wrap theme={null}
  Can you create a new launchdarkly project called serenia-agent-skills and also
  1. Create a new boolean feature flag
  2. Key: qualify-lead-skill
  3. Name: Qualify Lead Skill
  4. Set the default variation to true for test environment
  5. Make sure it's turned on in your environment
  ```
</CodeGroup>

Confirm `LD_SDK_KEY` in `.env` is the **server-side SDK key** for the `serenia-agent-skills` project, not a REST API access token.

<Frame caption="Creating the feature flag and project using the LaunchDarkly MCP Server in Cursor.">
  <img src="https://mintcdn.com/launchdarkly/AAfzx725JxtBqiAm/images/tutorials/datadog-to-launchdarkly-traces-part-one/create_feature_flag_and_project_using_cursor.png?fit=max&auto=format&n=AAfzx725JxtBqiAm&q=85&s=3be844d140eb8debec2ceae2945dd445" alt="Creating the feature flag and project using the LaunchDarkly MCP Server in Cursor." width="1760" height="2148" data-path="images/tutorials/datadog-to-launchdarkly-traces-part-one/create_feature_flag_and_project_using_cursor.png" />
</Frame>

Open **Flags** in the `serenia-agent-skills` project and verify the flag. This app evaluates the flag with the server-side Python SDK. You do not need to enable client-side SDK availability.

<Frame caption="The qualify-lead-skill feature flag in the LaunchDarkly UI.">
  <img src="https://mintcdn.com/launchdarkly/AAfzx725JxtBqiAm/images/tutorials/datadog-to-launchdarkly-traces-part-one/qualify_leades_feature_flag.png?fit=max&auto=format&n=AAfzx725JxtBqiAm&q=85&s=1f91868d0cafde740be57b51ca50b01f" alt="The qualify-lead-skill feature flag in the LaunchDarkly UI." width="1905" height="945" data-path="images/tutorials/datadog-to-launchdarkly-traces-part-one/qualify_leades_feature_flag.png" />
</Frame>

When the flag is **on**, `qualify_lead` runs. When it is **off**, the agent falls back to `log_inquiry`.

A custom SDK hook in [`serenia/observability/ld_hook.py`](https://github.com/arober39/serenia-agent-skills/blob/tutorial-datadog-part-one/serenia/observability/ld_hook.py) records each evaluation on the active Datadog span. After the flag is on, send another qualify-lead message. Open the trace in Datadog APM. The skill span includes tags like these:

<CodeGroup>
  ```title="Feature flag tags on Datadog spans" lines wrap theme={null}
  feature_flag.key:             "qualify-lead-skill"
  feature_flag.provider.name:   "LaunchDarkly"
  feature_flag.result.value:    "true"
  feature_flag.context.id:      "user-abc123"
  feature_flag.contextKeys:     {"user":"user-abc123"}
  ```
</CodeGroup>

<Frame caption="Datadog APM showing serenia-agent traces with the LaunchDarkly feature flag on the span.">
  <img src="https://mintcdn.com/launchdarkly/AAfzx725JxtBqiAm/images/tutorials/datadog-to-launchdarkly-traces-part-one/view-ld-feature-flag-in-DD-view-in-apm.png?fit=max&auto=format&n=AAfzx725JxtBqiAm&q=85&s=2113d5218acb3002d1355b71b305a598" alt="Datadog APM showing serenia-agent traces with the LaunchDarkly feature flag on the span." width="1024" height="795" data-path="images/tutorials/datadog-to-launchdarkly-traces-part-one/view-ld-feature-flag-in-DD-view-in-apm.png" />
</Frame>

## Forward traces to LaunchDarkly

Configure the Datadog Agent to dual-ship the same APM traces to LaunchDarkly. You do not change application code. Keep Datadog as the Agent's primary APM intake and add LaunchDarkly as an additional endpoint, as described in [Datadog Agent ingestion](/docs/home/observability/datadog-agent) and [Datadog dual shipping](https://docs.datadoghq.com/agent/configuration/dual-shipping/).

If the Agent is not installed yet, follow [Datadog's install guide](https://docs.datadoghq.com/agent/), then confirm it is running:

<CodeGroup>
  ```shell title="Check Datadog Agent status" lines wrap theme={null}
  sudo datadog-agent status
  ```
</CodeGroup>

Edit `datadog.yaml` (typically `/opt/datadog-agent/etc/datadog.yaml` on macOS or `/etc/datadog-agent/datadog.yaml` on Linux). Your `api_key` and `site` should already point at Datadog. Add this `apm_config` block:

<CodeGroup>
  ```yaml title="Dual shipping configuration in datadog.yaml" lines wrap theme={null}
  # /opt/datadog-agent/etc/datadog.yaml

  apm_config:
    enabled: true
    additional_endpoints:
      "https://datadog.observability.app.launchdarkly.com:8126":
        - placeholder
  ```
</CodeGroup>

<Note>
  The Datadog Agent requires a non-empty key for each additional endpoint. LaunchDarkly does not validate that value. Use any placeholder string.
</Note>

<Note>
  **LaunchDarkly only:** If you do not need traces in Datadog APM, set `apm_dd_url: https://datadog.observability.app.launchdarkly.com:8126` instead of `additional_endpoints`.
</Note>

Restart the Agent:

<CodeGroup>
  ```shell title="Restart the Datadog Agent" lines wrap theme={null}
  sudo datadog-agent restart
  ```
</CodeGroup>

Confirm `.env` still has `DD_AGENT_HOST=localhost` and `DD_TAGS=launchdarkly.project_id:` plus your client-side ID. Restart the Serenia API. After `ddtrace` reconnects, send a few messages.

In LaunchDarkly, go to **Observe > Traces** and click **Run query**. You should see traces from Datadog.

<Frame caption="Traces from Datadog appearing in the LaunchDarkly Observe Traces view.">
  <img src="https://mintcdn.com/launchdarkly/AAfzx725JxtBqiAm/images/tutorials/datadog-to-launchdarkly-traces-part-one/verify-traces-in-launchdarkly.png?fit=max&auto=format&n=AAfzx725JxtBqiAm&q=85&s=66be1b771641428e05bc59baab1548d1" alt="Traces from Datadog appearing in the LaunchDarkly Observe Traces view." width="1024" height="580" data-path="images/tutorials/datadog-to-launchdarkly-traces-part-one/verify-traces-in-launchdarkly.png" />
</Frame>

Expand the qualify-lead span. It includes the feature flag and, if CRM credentials are set, the Airtable record.

<Frame caption="Expanded qualify lead span in LaunchDarkly showing the feature flag, source, and Airtable record.">
  <img src="https://mintcdn.com/launchdarkly/AAfzx725JxtBqiAm/images/tutorials/datadog-to-launchdarkly-traces-part-one/LD-trace-qualify-leads-span.png?fit=max&auto=format&n=AAfzx725JxtBqiAm&q=85&s=70a4cd26296d062f7ca192b9fe5df387" alt="Expanded qualify lead span in LaunchDarkly showing the feature flag, source, and Airtable record." width="846" height="1024" data-path="images/tutorials/datadog-to-launchdarkly-traces-part-one/LD-trace-qualify-leads-span.png" />
</Frame>

## Further reading

* [LaunchDarkly Observability: Datadog Agent](/docs/home/observability/datadog-agent)
* [Datadog Agent Dual Shipping](https://docs.datadoghq.com/agent/configuration/dual-shipping/)