Already using Datadog? Here's how to gate and observe your AI agent skills with LaunchDarkly (without OpenTelemetry)

Sending Datadog trace data to LaunchDarkly.

If you’re already using Datadog to monitor your application, you can forward your existing application performance monitoring (APM) trace data to LaunchDarkly using the Datadog Agent. No additional instrumentation required. This unlocks feature flags, Guarded Rollouts, and automatic rollbacks that use Observability data from your existing Datadog configuration. This tutorial walks through that setup with Serenia, an AI assistant for an event space sales team. The higher-risk qualify_lead skill is gated by a feature flag. By the end, Datadog APM traces flow into both Datadog and LaunchDarkly. All code for this tutorial is in the tutorial-datadog-part-one branch of the Serenia repo.

Prerequisites

To complete this tutorial, you need:

A LaunchDarkly account with Writer permissions
A Datadog account with the Datadog Agent installed
An Anthropic API key
Cursor, with the LaunchDarkly Model Context Protocol (MCP) server configured
Python 3 and Node.js, to run the sample app
An optional Airtable account, if you want skills to write lead records

Serenia: Sales team AI assistant

When a customer sends a message about reserving an event space, the agent classifies their intent and routes it to a skill. Skills can write lead information to Airtable for sales follow-up.

The agent uses these skills:

Skill	Status	What it does
`answer_faq`	Stable	Answers venue questions using a knowledge base and Claude
`log_inquiry`	Stable	Records prospect contact info to Airtable
`qualify_lead`	Flag-gated	Scores leads as hot, warm, or cold and chooses a follow-up action
`auto_propose`	Locked	Generates custom event proposals (not used in this tutorial)

qualify_lead is heavier and higher-risk than log_inquiry. The qualify-lead-skill flag gates it. If the flag is off, the agent falls back to log_inquiry.

Clone the repo

To clone the demo and install dependencies:

git clone https://github.com/arober39/serenia-agent-skills.git
cd serenia-agent-skills
git checkout tutorial-datadog-part-one
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt

Set up credentials

Copy the example environment file and fill in your keys:

cp .env.example .env

You need credentials from these services:

Service	Variable(s)	Where to find it
Anthropic	`ANTHROPIC_API_KEY`	console.anthropic.com > API Keys
LaunchDarkly	`LD_SDK_KEY` (server-side SDK key), `LD_CLIENT_SIDE_ID`	Project settings > Environments
Datadog	`DD_API_KEY`, `DD_SITE`	Organization Settings > API Keys
Airtable	`AIRTABLE_PAT`, `AIRTABLE_BASE_ID`	airtable.com/create/tokens

For ddtrace, set DD_TAGS to your LaunchDarkly client-side ID. LaunchDarkly uses that tag to route traces to the correct project. Do not use OTEL_RESOURCE_ATTRIBUTES for this. Your .env should look like this:

ANTHROPIC_API_KEY=sk-ant-your-key
LD_SDK_KEY=sdk-your-key
LD_CLIENT_SIDE_ID=your-client-side-id
DD_API_KEY=your-datadog-api-key
DD_SITE=datadoghq.com
DD_AGENT_HOST=localhost
DD_SERVICE=serenia-agent
DD_TAGS=launchdarkly.project_id:YOUR_CLIENT_SIDE_ID
AIRTABLE_PAT=patYourToken
AIRTABLE_BASE_ID=appYourBaseId

Configure Airtable

Airtable is optional. If credentials are not set, skills still run and return responses. They skip the customer relationship management (CRM) write. The log_inquiry and qualify_lead skills write to a Leads table. Create these fields:

Field	Description
Name	Customer’s name
Email	Customer’s email address
Message	The customer’s original message
Status	`"New"` or `"Qualified"`
Lead Score	`"Hot"`, `"Warm"`, or `"Cold"`
Lead Action	`"Scheduled call"`, `"Sent brochure"`, or `"Sent product info"`
Qualification Reason	LLM-generated explanation of the score

The base ID is in your Airtable URL: airtable.com/<BASE_ID>/tbl. Generate a personal access token with data.records:write on your base. To create the base with the Airtable AI assistant, use this prompt:

Airtable AI assistant prompt

Create a base called "Serenia CRM" with a single table called "Leads" that has
  the following fields:

 - Name (Single line text)
 - Email (Email)
 - Message (Long text)
 - Status (Single select with options: New, Qualified)
 - Lead Score (Single select with options: Hot, Warm, Cold)
 - Lead Action (Single line text)
 - Qualification Reason (Long text)

 This table will be used to track customer inquiries and qualified leads for an
  event venue. The "log_inquiry" skill creates records with Status "New". The
 "qualify_lead" skill creates records with Status "Qualified" and fills in Lead
  Score, Lead Action, and Qualification Reason.

Using the Airtable AI assistant to create the Serenia CRM base.

Run the app and verify Datadog traces

The sample app is already instrumented with ddtrace. Start it, send a request, and confirm traces appear in Datadog APM before you forward anything to LaunchDarkly. Start the API:

uvicorn server:app --reload --port 8000

In a second terminal, start the UI:

cd ui
npm install
npm run dev

Generate traces by chatting with the agent at http://localhost:3000, or with curl:

curl -X POST http://localhost:8000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "Do you host baby showers?"}'

Open Datadog > APM > Traces and filter by service:serenia-agent. You should see traces for each request, with Anthropic calls nested inside skill spans such as qualify_lead or agent.handle_message.

Datadog APM traces for service:serenia-agent, with Anthropic calls nested under skill spans.

Those spans come from instrumentation already in the repo:

Auto-instrumentation: init_tracing() calls patch(anthropic=True) once at startup. Every Anthropic SDK call is traced without wrappers in skill code. Read serenia/observability/tracing.py.
Skill spans: trace_skill() creates a parent span that shows which skill ran, plus tags such as lead score and Airtable record ID. Read how qualify_lead.py uses it.

If Airtable credentials are set, new rows appear in Leads:

Airtable Leads table showing records created by both log_inquiry (Status: New) and qualify_lead (Status: Qualified, with Lead Score and Lead Action populated). — Airtable Leads table showing records created by log_inquiry and qualify_lead.

Create the qualify-lead feature flag

The app already checks a LaunchDarkly flag before qualify_lead runs. Create the flag. Evaluations then appear on Datadog spans when the skill runs. In Cursor, use the LaunchDarkly MCP server with this prompt:

Can you create a new launchdarkly project called serenia-agent-skills and also
Create a new boolean feature flag
Key: qualify-lead-skill
Name: Qualify Lead Skill
Set the default variation to true for test environment
Make sure it's turned on in your environment

Confirm LD_SDK_KEY in .env is the server-side SDK key for the serenia-agent-skills project, not a REST API access token.

Creating the feature flag and project using the LaunchDarkly MCP Server in Cursor.

Open Flags in the serenia-agent-skills project and verify the flag. This app evaluates the flag with the server-side Python SDK. You do not need to enable client-side SDK availability.

The qualify-lead-skill feature flag in the LaunchDarkly UI.

When the flag is on, qualify_lead runs. When it is off, the agent falls back to log_inquiry. A custom SDK hook in serenia/observability/ld_hook.py records each evaluation on the active Datadog span. After the flag is on, send another qualify-lead message. Open the trace in Datadog APM. The skill span includes tags like these:

feature_flag.key:             "qualify-lead-skill"
feature_flag.provider.name:   "LaunchDarkly"
feature_flag.result.value:    "true"
feature_flag.context.id:      "user-abc123"
feature_flag.contextKeys:     {"user":"user-abc123"}

Datadog APM showing serenia-agent traces with the LaunchDarkly feature flag on the span.

Forward traces to LaunchDarkly

Configure the Datadog Agent to dual-ship the same APM traces to LaunchDarkly. You do not change application code. Keep Datadog as the Agent’s primary APM intake and add LaunchDarkly as an additional endpoint, as described in Datadog Agent ingestion and Datadog dual shipping. If the Agent is not installed yet, follow Datadog’s install guide, then confirm it is running:

sudo datadog-agent status

Edit datadog.yaml (typically /opt/datadog-agent/etc/datadog.yaml on macOS or /etc/datadog-agent/datadog.yaml on Linux). Your api_key and site should already point at Datadog. Add this apm_config block:

# /opt/datadog-agent/etc/datadog.yaml

apm_config:
  enabled: true
  additional_endpoints:
    "https://datadog.observability.app.launchdarkly.com:8126":
      - placeholder

The Datadog Agent requires a non-empty key for each additional endpoint. LaunchDarkly does not validate that value. Use any placeholder string.

LaunchDarkly only: If you do not need traces in Datadog APM, set apm_dd_url: https://datadog.observability.app.launchdarkly.com:8126 instead of additional_endpoints.

Restart the Agent:

sudo datadog-agent restart

Confirm .env still has DD_AGENT_HOST=localhost and DD_TAGS=launchdarkly.project_id: plus your client-side ID. Restart the Serenia API. After ddtrace reconnects, send a few messages. In LaunchDarkly, go to Observe > Traces and click Run query. You should see traces from Datadog.

Traces from Datadog appearing in the LaunchDarkly Observe Traces view.

Expand the qualify-lead span. It includes the feature flag and, if CRM credentials are set, the Airtable record.

​Prerequisites

​Serenia: Sales team AI assistant

​Clone the repo

​Set up credentials

​Configure Airtable

​Run the app and verify Datadog traces

​Create the qualify-lead feature flag

​Forward traces to LaunchDarkly

​Further reading

Prerequisites

Serenia: Sales team AI assistant

Clone the repo

Set up credentials

Configure Airtable

Run the app and verify Datadog traces

Create the qualify-lead feature flag

Forward traces to LaunchDarkly

Further reading