Java AI SDK reference

This topic documents how to get started with the Java AI SDK, and links to reference information on all of the supported features.

The Java AI SDK is designed for use with AgentControl. You can follow development or contribute on GitHub.

Get started

LaunchDarkly AI SDKs interact with AgentControl configs. Configs are the LaunchDarkly resources that manage model configurations and messages for your generative AI applications.

You can use the Java AI SDK to customize your config based on the context you provide. This means both the messages and the model evaluation in your generative AI application are specific to each end user, at runtime. You can also use the Java AI SDK to record metrics from your AI model generation, including duration and tokens.

Follow these instructions to start using the Java AI SDK in your application.

Version compatibility

The Java AI SDK requires a minimum Java version of 8.

Install the SDK

This module is part of the java-core monorepo and is published to Maven Central as com.launchdarkly:launchdarkly-java-server-sdk-ai.

The Java AI SDK is built on top of the Java (server-side) SDK, so you need to install both packages. To find the latest version, refer to the SDK releases page.

Here is how:

1
<dependency>
2
  <groupId>com.launchdarkly</groupId>
3
  <artifactId>launchdarkly-java-server-sdk</artifactId>
4
</dependency>
5
<dependency>
6
  <groupId>com.launchdarkly</groupId>
7
  <artifactId>launchdarkly-java-server-sdk-ai</artifactId>
8
</dependency>

Initialize the client

After you install the SDK, create a single, shared instance of LDClient. When the LDClient is initialized, use it to initialize the LDAIClient. The LDAIClient is how you interact with configs.

Here is how:

1
LDClient ldClient = new LDClient(sdkKey);
2
LDAIClient aiClient = new LDAIClientImpl(ldClient);

Configure the context

Next, configure the context that will encounter generated AI content in your application. The context attributes determine which variation of the config LaunchDarkly serves to the end user, based on the targeting rules in your config. If you use template variables in the messages or instructions in your config’s variations, the context attributes also provide values for those variables.

Within a prompt message or agent instruction, the evaluation context is available as {{ldctx}}, for example {{ldctx.key}}.

Here is how:

1
LDContext context = LDContext.builder("example-context-key")
2
  .set("firstName", "Sandy")
3
  .set("lastName", "Smith")
4
  .set("email", "sandy@example.com")
5
  .build();

Customize a config

Customize your config to set values for variables used in messages or instructions, based on the context attributes and variables you provide.

The details of customizing a config depend on whether you are using configs in completion mode or agent mode. You set the mode when you create the config in the LaunchDarkly UI.

Customize configs in completion mode

In completion mode, each variation in your config includes a single set of roles and messages used to prompt your AI model.

Use completionConfig to customize the config. This method takes the following variables:

• AgentControl config key
• Context
• Fallback value
• Optional variables

It performs the evaluation, then returns an AICompletionConfig object that includes the customized model, provider, and messages, with variables already interpolated. Call createTracker on the returned object to get a tracker instance you can use to record metrics.

If completionConfig cannot perform the evaluation or LaunchDarkly is unreachable, the SDK returns the fallback configuration you provided. For example, you might use an empty, disabled AICompletionConfigDefault as a fallback value, or a fully configured default. Either way, check config.isEnabled() before using the result, and skip generation for the disabled case.

Here is how:

1
Map<String, Object> variables = new HashMap<>();
2
variables.put("username", "Sandy");
3
4
AICompletionConfig config = aiClient.completionConfig(
5
  "example-config-key",
6
  context,
7
  AICompletionConfigDefault.disabled(),
8
  variables
9
);
10
11
LDAIConfigTracker tracker = config.createTracker();

After you retrieve a config, config.getModel() and config.getProvider() are ready to pass to your model provider as-is. The config.getMessages() method returns the messages with any placeholders already interpolated.

Customize configs in agent mode

In agent mode, each variation in your config includes a set of instructions that enable multi-step workflows.

In agent mode, use agentConfig or agentConfigs to customize the config. agentConfig customizes a single agent config. agentConfigs customizes a list of them and returns a map keyed by agent key.

The SDK returns the instructions as you defined them for the variation in the LaunchDarkly UI. Call createTracker on the returned object to get a tracker instance for recording metrics.

If the method cannot perform the evaluation or LaunchDarkly is unreachable, it returns the fallback value. For example, you might use an empty, disabled AIAgentConfigDefault as a fallback value, or a fully configured default. Either way, check agent.isEnabled() before using the result, and skip generation for the disabled case.

Here is how:

1
Map<String, Object> variables = new HashMap<>();
2
variables.put("topic", "climate change");
3
4
AIAgentConfig agent = aiClient.agentConfig(
5
  "example-config-key",
6
  context,
7
  AIAgentConfigDefault.disabled(),
8
  variables
9
);

To learn more, read Customizing AgentControl configs.

Retrieve a judge config

Use judgeConfig to retrieve the judge configuration for a config variation. The method takes an AgentControl config key, a context, a fallback value, and optional variables. It returns an AIJudgeConfig object that includes the judge’s customized messages, model configuration, and the evaluation metric key to associate scores with.

To learn more about using a judge to score model output, read Evaluate output manually with a judge below.

Retrieve a config without interpolating placeholders

Use completionConfigTemplate, agentConfigTemplate, or judgeConfigTemplate to retrieve a config the same way as their non-template counterparts, but with Mustache placeholders, including {{ldctx.*}}, left intact. These methods omit the variables parameter, since interpolation is skipped. This is useful for displaying prompt previews, storing templates for later rendering, or auditing prompt content.

Evaluate output manually with a judge

The Java AI SDK does not automatically invoke judges on completion or agent calls. To evaluate output yourself:

1. Retrieve the judge config with judgeConfig.
2. Implement Runner, a single-method interface that wraps whatever model provider SDK you use, so the SDK can invoke your model when it runs a judge prompt. The SDK does not ship provider-specific Runner implementations, so you write your own for whatever provider you use.
3. Construct a Judge by passing the judge config, your Runner implementation, and a logger to the Judge constructor.
4. Call evaluate on the Judge to score an input and output pair. The evaluate method takes the input, the output, and an optional sampling rate, and returns a JudgeResult with the score and reasoning.
5. Record the result with the tracker’s trackJudgeResult method.

Here is how:

1
AIJudgeConfig judgeConfig = aiClient.judgeConfig(
2
  "example-judge-key",
3
  context,
4
  AIJudgeConfigDefault.disabled(),
5
  variables
6
);
7
8
if (judgeConfig.isEnabled()) {
9
  Judge judge = new Judge(judgeConfig, runner, logger);
10
  JudgeResult result = judge.evaluate(inputText, outputText);
11
12
  // Record the result against the tracker for the completion or agent config
13
  // whose output you are evaluating, so the score is associated with that run.
14
  tracker.trackJudgeResult(result);
15
}

Manual judge evaluation does not automatically emit metrics for the config’s Monitoring tab. It also does not attach judges to variations in the LaunchDarkly UI.

To learn more, read Online evaluations in AgentControl and Judges.

Call the provider and record metrics from AI model generation

To make a request to your generative AI provider and record metrics from the response:

1. Check config.isEnabled() before generating, and skip generation for the disabled case.
2. Create a tracker with config.createTracker().
3. Wrap your provider call in trackDurationOf to record its duration. Alternatively, call the individual track* methods directly, such as trackTimeToFirstToken, trackToolCall, or trackFeedback.
4. Record token usage with trackTokens.
5. Record the outcome with trackSuccess or trackError.

Here is how:

1
if (config.isEnabled()) {
2
  LDAIConfigTracker tracker = config.createTracker();
3
4
  Response response = tracker.trackDurationOf(() -> {
5
    // Make a request to your generative AI provider, using details from
6
    // the config. For example, pass config.getModel() and config.getMessages().
7
    return callProvider();
8
  });
9
10
  tracker.trackTokens(new TokenUsage(response.getTotalTokens(), response.getInputTokens(), response.getOutputTokens()));
11
  tracker.trackSuccess();
12
}

Call completionConfig or agentConfig each time you generate content from your AI model, and call createTracker each time to start a new run.

To learn more, read Tracking AI metrics.

Agent graphs

Agent graphs let you define a directed graph of agent configs in the LaunchDarkly UI, then traverse the graph at runtime to orchestrate multi-step workflows. Each node in the graph is an agent config. Each edge defines a handoff from one agent to the next.

Use agentGraph to retrieve a graph by its key. The graph key populates automatically when you create a graph in the LaunchDarkly UI. After the graph exists, the key is in the agent graph details panel. To learn more, read Initialize an agent graph.

To set up a graph invocation:

1. Call agentGraph with the graph key, a context, and any variables. The method returns an AgentGraphDefinition that exposes the graph’s topology and a tracker for recording invocation metrics.
2. Check graph.isEnabled() before walking the graph.
3. Create a tracker with graph.createTracker() to record invocation metrics.

Then walk the graph manually or with traverse/reverseTraverse, as described below.

Here is how:

1
Map<String, Object> variables = new HashMap<>();
2
variables.put("topic", "climate change");
3
4
AgentGraphDefinition graph = aiClient.agentGraph("example-graph-key", context, variables);
5
6
if (graph.isEnabled()) {
7
  AIGraphTracker tracker = graph.createTracker();
8
  // Walk the graph and invoke each agent's config.
9
}

You can navigate the graph manually using rootNode, getNode, getChildNodes, getParentNodes, and terminalNodes. Each AgentGraphNode exposes:

• getKey(): The agent config key for this node.
• getConfig(): The AIAgentConfig for the node, with its instructions, model, tools, and other agent config properties.
• getEdges(): The outgoing GraphEdge list from this node. Each edge includes the target node key and a handoff map describing how to pass state to the next node.
• isTerminal(): Whether this node has no outgoing edges.

Alternatively, use traverse or reverseTraverse to walk the graph with a callback that the SDK invokes for each node. traverse walks the graph in topological order starting from the root, visiting a node only after all of its reachable predecessors have been visited. reverseTraverse walks in topological order starting from the terminal nodes, visiting a node only after all of its reachable descendants have been visited, so the root is visited last.

Both methods take a callback with the signature BiFunction<AgentGraphNode, Map<String, Object>, Object> and an initial context map. The SDK calls the callback one time for each node, passing the node and a context map scoped to that node’s dependency results. The callback’s return value is stored under the node’s key for downstream nodes to read.

Here is how:

1
graph.traverse((node, ctx) -> {
2
  // Use node.getConfig() to call your AI provider, passing instructions and tools.
3
  // Read upstream results from ctx as needed.
4
  Object output = callModelForNode(node.getConfig(), ctx);
5
6
  // The return value is stored under node.getKey() for downstream nodes.
7
  return output;
8
}, new HashMap<>());

The SDK does not automatically record handoff or invocation metrics during traversal. Call the appropriate AIGraphTracker methods from your callback to record what happens at each node and edge.

Use the AIGraphTracker returned by createTracker to record metrics for the full graph invocation as well as per-edge handoffs:

• trackInvocationSuccess and trackInvocationFailure record whether the graph completed successfully.
• trackDuration records total invocation duration.
• trackTotalTokens records aggregate token usage across all nodes.
• trackPath records the sequence of nodes visited.
• trackHandoffSuccess and trackHandoffFailure record per-edge handoff outcomes.
• trackRedirect records when a node redirects to a different target than its declared edges.

You can call the provider for each node the same way you do in agent mode. To learn more, read Customize configs in agent mode.

To learn more about defining agent graphs in the LaunchDarkly UI, read Agent graphs.

Resume tracking across processes

If a single AI workflow spans multiple processes, you can resume tracking on the same config or graph invocation in a different process. For example, a request handler might hand work off to a background job:

1. In the original process, create the config and tracker as usual.
2. Call getResumptionToken on the tracker to get a token.
3. Persist or pass the token to the next process, through a job queue, database row, or message envelope.
4. In the receiving process, call aiClient.createTracker for a completion or agent tracker, or aiClient.createGraphTracker for an agent graph tracker, passing the token and a context. Both methods return a tracker that records to the same invocation as the original.

Here is how:

1
AICompletionConfig config = aiClient.completionConfig("example-config-key", context, fallbackConfig, variables);
2
LDAIConfigTracker tracker = config.createTracker();
3
4
// Hand off work to another process. Persist the resumption token so the
5
// receiving process can reconstruct the tracker.
6
String token = tracker.getResumptionToken();
7
enqueueBackgroundJob(token, context);

For agent graphs, use createGraphTracker in place of createTracker. The token comes from AIGraphTracker.getResumptionToken() on the original graph tracker.

Supported features

This SDK supports the following features:

• Anonymous contexts
• Context configuration
• Customizing AgentControl configs
• Private attributes
• Tracking AI metrics