Python AI SDK reference

This topic explains how to get started with the Python AI SDK, and links to reference information on all of the supported features. This SDK is the current AI SDK for Python. It is not a drop-in replacement for the earlier legacy Python AI SDK, which is now in maintenance mode. We recommend that new applications use this SDK. If you are migrating from the legacy SDK, read Migrate from the legacy AI SDKs.

The Python AI SDK provides an opinionated, abstracted entry point to AgentControl. Rather than configuring provider routing, client wiring, and message conversion yourself, the SDK handles them for you.

Get started

LaunchDarkly AI SDKs interact with AgentControl configs. Configs are the LaunchDarkly resources that manage model configurations and messages for your generative AI applications. When you invoke a config with the Python AI SDK, LaunchDarkly controls which model, provider, prompt, and tools are used at runtime. You change providers, models, or prompts in LaunchDarkly without deploying code.

The Python AI SDK follows a three-tier architecture:

1. The core client (launchdarkly-ai-server) manages the LaunchDarkly lifecycle, telemetry, and the config(), graph(), and resolve_graph() entry-point functions.
2. Handler packages integrate a specific provider and mode, such as OpenAI in messages mode or Anthropic in agent mode.
3. Your application registers the handlers and tools it needs, then makes calls.

Follow these instructions to start using the Python AI SDK in your application.

Prerequisites

To use the Python AI SDK, you need:

• Python 3.12 or later.
• A LaunchDarkly account where you have completed the Quickstart for AgentControl.
• An existing AgentControl config.

Install the SDK

The SDK installs in two parts: a single core package that every application needs, and one handler package for each provider and mode you plan to use.

First, install the Python AI SDK by adding launchdarkly-ai-server. This package includes the core client and the base LaunchDarkly Python SDK, so you don’t need to install them separately.

Then, install a handler package for each provider and mode you want to use. The following handler packages are available:

If your provider does not have a pre-built package, use the LangChain handler package, which works with any provider, or write a custom handler.

Here’s how to install the LaunchDarkly core package and the OpenAI messages handler:

$
pip install launchdarkly-ai-server launchdarkly-ai-openai-messages

Replace launchdarkly-ai-openai-messages with the handler package for your provider from the table above.

Edge and custom runtimes

The launchdarkly-ai-server package bundles the base Python server SDK, which doesn’t run on every platform. If your platform restricts package installation or limits execution time, such as an edge or serverless runtime, install the unbundled core launchdarkly-ai-server package instead and pair it with the LaunchDarkly SDK for your platform. Then call init_client(pre_initialized_client) with your platform’s client before your first AI call.

To learn more, read Manage the client lifecycle.

Configure the environment

The SDK reads your LaunchDarkly SDK key and your provider API keys from environment variables. Set LD_SDK_KEY and the API key for each provider you use, such as OPENAI_API_KEY or ANTHROPIC_API_KEY.

The SDK also reads optional environment variables for connection routing, including the following:

• LD_BASE_URI
• LD_STREAM_URI
• LD_EVENTS_URI

It also reads optional variables for observability. To learn more about observability variables, read the Configure observability section.

Your SDK key is specific to each project and environment. Find it by going to your LaunchDarkly Settings and locating the SDK keys page.

How you set these depends on your setup. Common options include a .env file, your shell, or your platform’s environment configuration UI. Many secrets managers can inject stored values as environment variables too.

The examples that follow use a .env file loaded with load_dotenv().

Call a model with a convenience function

A convenience function is a single, pre-wired call. It’s the most opinionated entry point and intentionally trades configurability for speed. You can swap a provider SDK call for the LaunchDarkly equivalent and get the same behavior as well as targeting, metrics, and prompt management.

Each handler package exports one convenience function, the shortest path to a working call. Import it and call it with the user input, an LDContext, and the AgentControl config key. Behind the scenes, the SDK evaluates the config, selects the provider, calls the model, and records metrics.

Here’s how to call an OpenAI config:

1
from dotenv import load_dotenv
2
from launchdarkly_ai_openai_messages import openai_messages
3
4
load_dotenv()
5
6
result = await openai_messages(
7
    "What is feature flagging?",                # user input
8
    {"kind": "user", "key": "user-123"},    # user context: identifies who is making this request
9
    {"key": "my-ai-config-flag"},              # AgentControl config key: which LD config to use
10
    { "my_variable": "my-value" }.            # Optional: variables to interpolate into your messages
11
    [{ role: "assistant", content: "some assistant message" }] Optional: conversation history
12
)
13
14
print(result.response)

Call a model with a handler

For more control, use config() from the core package. Unlike a convenience function, config() gives you routing across multiple providers, custom variables, tools, streaming, and async judges. The config() function accepts an AgentControl config key and one or more handlers, then routes to the correct handler at invocation time based on the config variation’s provider and mode. It returns an object with the invoke() and stream() methods.

Set the output format on the config in the LaunchDarkly UI. When an output format is set, the SDK applies it to your call automatically, and invoke() returns a parsed object as response instead of a string.

Call invoke() with the user input, an LDContext, and an optional map of variables. The SDK uses the variables to fill in {{variable}} placeholders in the config’s messages or instructions. It also injects the LDContext under the ld_context key, so templates can reference context attributes with {{ld_context.key}} or {{ld_context.email}}.

Here’s how to route between multiple providers:

1
from dotenv import load_dotenv
2
from launchdarkly_ai_server import config, shutdown
3
from launchdarkly_ai_openai_messages import create_openai_messages_handler
4
from launchdarkly_ai_openai_agents import create_openai_agent_handler
5
from launchdarkly_ai_claude_messages import create_claude_messages_handler
6
from launchdarkly_ai_claude_agents import create_claude_agents_handler
7
from launchdarkly_ai_langchain_messages import create_langchain_messages_handler
8
9
load_dotenv()
10
11
result = await config(
12
    key="my-ai-config-flag",   # AgentControl config key
13
    handler=[                   # handlers, most specific first; wildcard fallback last
14
        create_openai_messages_handler(),
15
        create_openai_agent_handler(),
16
        create_claude_messages_handler(),
17
        create_claude_agents_handler(),
18
        create_langchain_messages_handler(),
19
    ],
20
).invoke(
21
    "What is feature flagging?",           # user input
22
    {"kind": "user", "key": "user-123"},   # user context
23
    {"user_name": "Ada"},                  # optional variables for the prompt template
24
)
25
26
print(result.response)
27
28
# Flush pending LaunchDarkly events and OpenTelemetry spans, then close the connection
29
await shutdown()

Write a custom handler

If you use a provider that does not have a pre-built package, or you want an internal endpoint, use create_handler() to build your own. The create_handler(provides_for, fn) function attaches routing metadata to a function so it works with config() routing and registries.

You can also use a custom handler as a fallback for providers or models you do not support. Register a handler that returns an unsupported-provider response, and config() routes unmatched calls to it.

Here’s how to write a custom handler:

1
import httpx
2
from launchdarkly_ai_server import config, create_handler
3
4
async def _call_internal(ld_config, user_input, tool_handlers, variables):
5
    # The SDK passes in ld_config (the resolved LD config) and user_input (the caller's request).
6
    async with httpx.AsyncClient() as client:
7
        response = await client.post(
8
            "https://models.internal.example.com/generate",
9
            json={"model": ld_config.model.name, "prompt": user_input},
10
        )
11
    data = response.json()
12
    return {"output": data["text"], "usage": data["usage"]}
13
14
internal_handler = create_handler(["InternalProvider", "messages"], _call_internal)
15
16
result = await config(
17
    key="my-ai-config-flag",
18
    handler=internal_handler,
19
).invoke("What is feature flagging?", {"kind": "user", "key": "user-123"})

Manage handlers and tools with a registry

Rather than passing handlers and tools to every call, you can register them once and reuse them. A Registry is a container with two properties:

• handlers: A list of handler objects created by provider handler functions.
• tools: A dictionary whose keys are tool names and whose values are the functions that run when the model calls each tool.

A registry can hold tools for many different configs. When you use a registry with config(), the SDK only passes the tools that the resolved config variation names, not every tool in the registry. The registry defines which tools are available for selection, not which tools an individual call receives.

You pass the registry as the registry option to any call.

Use a scoped registry

We recommend using scoped registries over the global registry, because they follow the principle of least privilege. Each registry exposes only the handlers and tools a given call site needs. Use a scoped registry when different parts of your application need different handlers or tools.

Here’s how to create and use a scoped registry:

1
from launchdarkly_ai_server import Registry, config
2
from launchdarkly_ai_openai_messages import create_openai_messages_handler
3
from launchdarkly_ai_openai_agents import create_openai_agent_handler
4
5
openai_registry = Registry(
6
    handlers=[create_openai_messages_handler(), create_openai_agent_handler()],
7
    tools={"get-prefs": get_preferences_fn},
8
)
9
10
result = await config(key="my-ai-config-flag", registry=openai_registry).invoke(user_input, context)

Use the global registry

The global_registry is a process-wide singleton exported from the core package. Populate it one time at startup, then reference it from any call site. Because it is process-wide, use it only when a single set of handlers and tools is appropriate for your whole application.

Here’s how to populate and use the global registry:

1
from launchdarkly_ai_server import global_registry, config
2
from launchdarkly_ai_openai_messages import create_openai_messages_handler
3
from launchdarkly_ai_claude_agents import create_claude_agents_handler, ClaudeWebSearch
4
5
# Call once at application startup
6
global_registry.register(
7
    handlers=[create_openai_messages_handler(), create_claude_agents_handler()],
8
    tools={
9
        "web-search": ClaudeWebSearch,
10
        "get-prefs": get_preferences_fn,
11
    },
12
)
13
14
# Reference the registry from any call site
15
result = await config(key="my-ai-config-flag", registry=global_registry).invoke(user_input, context)

You can combine two registries with compose(a, b). For example, you might layer call-site-specific handlers and tools on top of a shared base registry rather than redefining them. It returns a new third registry and does not modify either input.

When a handler key or tool name appears in both, the second registry takes precedence, so later registries override earlier ones. compose() works on any two registries, whether scoped, global, or a mix.

You can also pass handler and tool_handlers inline on a single call, even when you supply a registry. Inline options always take precedence over the registry, so you can override a handler or tool for one call without changing shared configuration.

Use built-in provider tools

Some provider SDKs expose sentinels, or built-in capabilities, such as web search in the Claude Agent SDK, that the provider handles natively rather than dispatching to a function you write. To use one, assign the built-in tool directly as the tool implementation in place of a function. For example, assign ClaudeWebSearch as the implementation for a tool key, and the SDK uses Claude’s built-in web search instead of dispatching to code you write. Assign LaunchDarkly tool keys to these built-ins so you can use them with LaunchDarkly features such as tool-call tracking.

Here’s how to enable the Claude web search tool:

1
from launchdarkly_ai_server import config
2
from launchdarkly_ai_claude_agents import create_claude_agents_handler, ClaudeWebSearch
3
4
result = await config(
5
    key="my-ai-config-flag",
6
    tool_handlers={
7
        "web-search": ClaudeWebSearch,           # built-in tool, no function needed
8
        "get-prefs": get_preferences_fn,
9
    },
10
    handler=[create_claude_agents_handler()],
11
).invoke("What are the latest LaunchDarkly release notes?", {"kind": "user", "key": "user-123"})

The launchdarkly-ai-claude-agents package exports additional built-in tools beyond web search, such as file and shell operations. For the current set, read the package documentation for your preferred provider.

Run an agent graph

An agent graph is a multi-agent workflow defined in a LaunchDarkly graph flag as a root agent plus directed edges. Use the graph() function to run one.

The SDK uses a model-driven router. It starts at the root node and presents the outgoing edges to the model as handoff choices. When the model selects an edge, the router follows it, passing the original user request and the previous node’s response to the next node.

The loop ends when any of these conditions is met:

• The model produces a final answer.
• Execution reaches a leaf node, a node with no outgoing edges where the graph ends.
• The router detects a cycle.
• Execution hits the step cap.

Each node runs through the same path as config(), so every node emits its own telemetry and runs its own judges, tagged with the graph key. Because routing is per node, you can build mixed graphs in which different nodes use different providers.

Here’s how to run a graph:

1
from dotenv import load_dotenv
2
from launchdarkly_ai_server import graph, shutdown
3
from launchdarkly_ai_claude_agents import create_claude_agents_handler
4
from launchdarkly_ai_openai_agents import create_openai_agent_handler
5
6
load_dotenv()
7
8
result = await graph(
9
    "support-graph", 
10
    handlers=[create_claude_agents_handler(), create_openai_agent_handler()],
11
).invoke(
12
    "I was double charged",
13
    {"kind": "user", "key": "user-123"},
14
    {"account_tier": "pro"},
15
)
16
17
print(result["response"])  # final output
18
print(result["usage"])     # aggregate token counts
19
20
await shutdown()

To evaluate the final graph output with a judge, pass a judge config key as the graph_judge option. Use it when you want a single quality or safety check on the graph’s final answer, separate from any judges that run on individual nodes. The handler packages also export single-provider convenience functions: claude_graph, openai_graph, and langchain_graph. Each pre-binds its own handler and runs the graph using that provider’s own agent or graph SDK, rather than the model-driven router described above.

For example, openai_graph runs using the OpenAI Agents SDK’s own harness. To learn more about running graphs on a provider’s own framework, read Run a graph on a native framework runner. For mixed-provider graphs, or when you want the router’s model-driven handoff behavior, use the base graph() function and pass multiple handlers.

Run a graph on a native framework runner

If you want a provider’s own framework to manage handoffs, tool loops, and conversation state, use the resolve_graph() function to resolve the topology without executing it, then pass the result to a native runner. Each handler package ships a runner that converts the resolved graph into the native agent and handoff constructs the provider’s framework uses.

The following native runners are available:

Here’s how to run a graph on LangGraph:

1
from launchdarkly_ai_server import resolve_graph
2
from launchdarkly_ai_langchain_agents import to_lang_graph
3
4
result = await to_lang_graph(
5
    resolve_graph("support-graph", context=context),  # 'context=' is the parameter; the value is your LDContext variable
6
    tool_handlers=registry.tools,
7
    context=context,
8
).ainvoke("I was double charged")

The .ainvoke() method is LangGraph’s own async method, not a LaunchDarkly function. The a prefix is LangGraph’s convention for async calls.

Native runners bypass the SDK’s model-driven router, but they still accept the same tool_handlers map as graph(), emit the same $ld:ai:* tracking events, and wrap execution in a parent OpenTelemetry span.

Use structured output

A config can return structured output that conforms to a JSON schema, rather than a plain string.

Use structured output when:

• The response needs to be machine-readable, not just human-readable.
• You feed model output into another system, database, or function.
• You need to extract specific structured information, such as a severity or category.
• You build pipelines where the output of one call feeds the input of another.

Set the output format on the config in the LaunchDarkly UI. When an output format is set, the SDK applies it to your call automatically, and invoke() returns a parsed object as response instead of a string.

For providers with native structured output support, such as OpenAI in messages mode, the SDK uses the provider’s built-in structured output feature. For all other providers, the SDK appends schema instructions to the prompt on a best-effort basis. If the model returns output that cannot be parsed, invoke() returns the raw string rather than raising an exception.

Stream a response

Streaming delivers the model’s response incrementally as it generates, rather than waiting for the complete response.

To receive a response token by token, use stream() instead of invoke(). The stream() method returns an async generator that yields a chunk event for each text delta, followed by a single done event that carries the full response, normalized token usage, and any judge results.

Here’s how to stream a response:

1
from launchdarkly_ai_server import config, global_registry
2
3
async for event in config(key="my-ai-config-flag", registry=global_registry).stream(
4
    "What is feature flagging?",
5
    {"kind": "user", "key": "user-123"},
6
):
7
    if event["type"] == "chunk":
8
        print(event["text"], end="", flush=True)
9
    else:
10
        # Final event: full response, usage, and judge results.
11
        print("\n", event["usage"])

Run judges asynchronously

Judges evaluate a model’s output against criteria you define in a judge config. By default, judges configured on a config run inline, which adds their evaluation time to your response latency. To run judges without increasing latency, evaluate them in a background task.

Pass skip_judges=True to config() to suppress inline evaluation. invoke() then returns judge_tasks, a list of pre-packaged, serializable tasks, alongside the model response. Schedule each task on a background worker (a task that runs independently without blocking your main application) and call run_judge(task, handlers) inside the worker. Because each task carries everything the judge needs, the worker handles the AI call and the LaunchDarkly tracking event on its own, and your main thread does not block.

You can use any async mechanism you choose. This example uses asyncio.create_task, but a thread, a task queue, or an external worker such as a Lambda works too.

Here’s how to return judge tasks and hand them to background workers:

1
import asyncio
2
from launchdarkly_ai_server import config, global_registry
3
4
result = await config(
5
    key="my-ai-config-flag",
6
    registry=global_registry,
7
    skip_judges=True,
8
).invoke(user_input, context)
9
10
# Fire and forget: the main coroutine continues immediately
11
for task in result.judge_tasks or []:
12
    asyncio.create_task(run_judge_worker(task))

Configure observability

The Python AI SDK records AI metrics, such as duration and token counts, automatically on every call. It also creates OpenTelemetry spans that follow the Gen AI semantic conventions. You do not need to initialize a tracker or wrap your model calls in spans yourself.

To export traces to LaunchDarkly Observability, make the OpenTelemetry SDK packages available at runtime. You have two options:

• Install the otel extra with pip install "launchdarkly-ai[otel]", which bundles the OpenTelemetry packages the client needs. The client discovers them automatically.
• Configure your own OpenTelemetry setup. The client uses the OpenTelemetry packages already present in your environment.

At runtime, the client detects whether the OpenTelemetry packages are present. If they are, it configures a tracer provider with a GZIP-compressed OTLP HTTP exporter and W3C trace-context propagators. If they are absent, it logs a one-time warning with the install command and continues. Feature flags and AI calls work normally, and spans become no-ops.

The following environment variables configure telemetry:

To learn more, read LLM observability and Monitor configs.

Manage the client lifecycle

The Python AI SDK manages a single shared LaunchDarkly client for you. In most applications, the client initializes lazily on your first AI call, so you do not need to call init_client() explicitly.

Call init_client() yourself when you want to do the following:

• Pass custom initialization options.
• Run in an edge or custom runtime and supply a pre-initialized client. Call init_client(pre_initialized_client) with a client from your platform’s LaunchDarkly SDK before your first AI call.
• Pre-warm the client rather than on the first request. Pre-warming initializes it at startup so it’s ready before your first user request arrives. We recommend pre-warming the client when first-request latency matters, such as for predictable traffic spikes or serverless cold starts.

Call shutdown() when your application exits to flush pending LaunchDarkly events and OpenTelemetry spans and close the connection. shutdown() is safe to call more than once.

Supported features

The Python AI SDK builds on the Python SDK and supports the following features:

• Anonymous contexts
• Context configuration
• Customizing AgentControl configs
• Private attributes
• Tracking AI metrics

What’s next

Now that you can call models with the Python AI SDK, explore these next steps:

• Create configs to set up your own AgentControl configs in the LaunchDarkly UI.
• Agent graphs to build multi-agent workflows.