Managing AI model configuration outside of code with the Node.js AI SDK

Overview

This guide shows how to manage AI model configuration and prompts for an OpenAI-powered application in a runtime environment. It uses the LaunchDarkly Node.js (server-side) AI SDK and AgentControl to dynamically customize your application.

Using AgentControl to customize your application means you can:

• manage your model configuration and prompts outside of your application code
• enable non-engineers to iterate on prompt and model configurations
• apply updates to prompts and configurations without redeploying your application

This guide steps you through the process of working in your application and in LaunchDarkly to customize your application.

If you’re already familiar with setting up AgentControl in the LaunchDarkly UI and want to skip straight to the sample code, you can find it on GitHub.

Prerequisites

To complete this guide, you must have the following prerequisites:

• a LaunchDarkly account, including
  • a LaunchDarkly SDK key for your environment.
  • a role that allows AgentControl actions. The LaunchDarkly Project Admin, Maintainer, and Developer project roles, as well as the Admin and Owner base roles, all include this ability.
• a Node.js (server-side) application. This guide provides sample code in TypeScript. You can omit the types if you are using JavaScript.
• an OpenAI API key. The LaunchDarkly AI SDKs provide specific functions for completions for several common AI model families, and an option to record this information yourself. This guide uses OpenAI.

Example scenario

In this example, you manage a product recommendation system for an e-commerce platform. Using the LaunchDarkly Node.js (server-side) AI SDK, you’ll configure AI prompts to provide personalized product suggestions based on your customers’ preferences. You’ll also track metrics, such as the number of output tokens used by your generative AI application.

Step 1: Prepare your development environment

First, install the required SDKs:

$
npm install @launchdarkly/node-server-sdk "@launchdarkly/server-sdk-ai@^0.20.0" "@launchdarkly/server-sdk-ai-openai@^0.6.0"

Then, set up credentials in your environment:

$
export OPENAI_API_KEY=your_openai_api_key
$
export LAUNCHDARKLY_SDK_KEY=your_launchdarkly_sdk_key

If you are starting from scratch, you can use the Node dotenv package in conjunction with a .env file to load your API keys.

Step 2: Initialize LaunchDarkly SDK clients

Inside of your project, create a shared utility for initializing the LaunchDarkly SDK and its AI client.

Create a config/launchDarkly.ts file:

1
import { init, LDClient } from "@launchdarkly/node-server-sdk";
2
import { initAi, LDAIClient } from "@launchdarkly/server-sdk-ai";
3
4
const LD_SDK_KEY = process.env.LAUNCHDARKLY_SDK_KEY;
5
6
const ldClient: LDClient = init(LD_SDK_KEY as string);
7
8
const aiClient: LDAIClient = initAi(ldClient);
9
// Initialize and return the LaunchDarkly client
10
export const getLaunchDarklyClients = async () => {
11
  try {
12
    await ldClient.waitForInitialization({ timeout: 10 });
13
  } catch (err) {
14
    // log your error
15
  }
16
  return { ldClient, aiClient };
17
};

Step 3: Set up AgentControl configs in LaunchDarkly

Next, create an AgentControl config in the LaunchDarkly UI. AgentControl configs are the LaunchDarkly resources that manage model configurations and messages for your generative AI applications.

To create an AgentControl config:

1. In LaunchDarkly, click Create and choose AgentControl config.
2. In the “Create AgentControl config” dialog, give your AgentControl config a human-readable Name, for example, “chat-helper-v1” or “shopping assistant.”
3. Click Create.

Then, configure the model and prompt by creating a variation. Every AgentControl config has one or more variations, each of which includes your AI messages and model configuration.

Here’s how:

1. In the create panel in the Variations tab, replace “Untitled variation” with a variation Name. You’ll use this to refer to the variation when you set up targeting rules, below.
2. Click Select a model and select a supported OpenAI model, for example, gpt-4o.
3. Optionally, adjust the model parameters: click Parameters to view and update model parameters. In the dialog, adjust the model parameters as needed. The Base value of each parameter is from the model settings. You can choose different values for this variation if you prefer.

Next, add system, user, or assistant messages to define your prompt. In this example, you’ll augment the prompt with saved user preferences.

Start with the system message:

You are an e-commerce shopping assistant. You help users find products that match their preferences.

Now, add an assistant message that pre-loads the model with the user’s preferences and provides some instructions about the input format, response format, and context:

Products are listed in the following format: [Product Name, Product ID, Product Type, {Preference tags}]
The user's preferences are {{preferences}}
The products available for purchase are: {{productsAvailable}}
Return the matched products as a list in the following format:
- [productName, productId]
- [productName, productId]
Address the response to {{ldctx.name}}

The double curly braces in the prompts allow you to augment the messages with customized data at runtime. To learn more, read Syntax for customization.

Within the UI, the variation will look like the following:

Click Review and save at the top right of the page to save the configuration.

Step 4: Enable targeting in LaunchDarkly

Targeting is how you specify that specific end users working in your application will receive specific variations that you’ve created in your AgentControl config.

To set up targeting, click the Targeting tab. Targeting for AgentControl configs is on by default. Set the default targeting rule to serve the “Shopping preference assistant” variation you just created for your test environment.

To specify the AgentControl config variation to use by default when the AgentControl config is toggled on:

1. Select the Targeting tab for your AgentControl config.
2. In the “Default rule” section, click Edit.
3. Set the default rule to serve the shopping preference assistant variation.
4. Click Review and save. In the confirmation dialog, click Save changes.

By default, this AgentControl config will now serve the variation shopping preference assistant variation:

Step 5: Integrate the AgentControl config into your application

To integrate the LaunchDarkly AgentControl config into your application, you’ll need to set up your OpenAI client, set up your application data, add your completion code, and finally put it all together.

Set up the OpenAI client

Next, set up your OpenAI client. Following a similar pattern as above, create a config/openAi.ts file to initialize the OpenAI client:

1
import { OpenAI } from "openai";
2
3
let openaiClient: OpenAI;
4
5
export const getOpenAI = () => {
6
    if(!openaiClient) {
7
        openaiClient = new OpenAI();
8
    }
9
    return openaiClient;
10
}

Set up application data

Then, set up the data for your application. Normally, the product data that your application retrieves would come from a database. For this example, create a data/products.ts file to serve mock data:

1
import { productsAvailable } from "./mockProducts";
2
3
export type Product = {
4
    name: string;
5
    productId: string;
6
    description: string;
7
    price: number;
8
    type: "shoes" | "clothing" | "accessories";
9
    preferenceTags: string[];
10
}
11
12
export const getProducts = () => {
13
    return {
14
        productsAvailable,
15
    }
16
}
17
18
// format is [Product Name, Product ID, Product Type, {Preference tags}]
19
export const formatProductForCompletion = (product: Product) => {
20
    return `[${product.name}, ${product.productId}, ${product.type}, ${product.preferenceTags.join(", ")}]`
21
}

Additionally, add the following data/mockProducts.ts file in the same folder:

1
import { Product} from "./products"
2
3
export const productsAvailable: Product[] = [
4
    {
5
        name: "AeroGlide Elite Runner 3.0",
6
        productId: "1",
7
        description: "Versatile running shoes with responsive cushioning",
8
        type: "shoes",
9
        price: 120,
10
        preferenceTags: ["athletic", "running", "comfort"]
11
    },
12
    // complete list of products available in linked file
13
  ]

The data contains the name, price, type, preference tags to match to a customer’s preferences, a description, and a product ID. This should give the AI model ample data to match to a customer’s query.

Add your completion

Next, add the AgentControl config and completion code into your application. Create a completions/shoppingAssistant.ts file:

1
import { LDContext } from "@launchdarkly/node-server-sdk";
2
import { getLaunchDarklyClients } from "../config/launchdarkly";
3
import { getOpenAI } from "../config/openAI";
4
import { formatProductForCompletion, getProducts } from "../data/products";
5
import { LDFeedbackKind } from "@launchdarkly/server-sdk-ai";
6
import { getAIMetricsFromResponse } from "@launchdarkly/server-sdk-ai-openai";
7
8
export const completeShoppingAssistant = async (userName: string, userMessage: string, userPreferences: string[]) => {
9
  const openaiClient = getOpenAI();
10
11
  const { productsAvailable } = await getProducts();
12
13
  const { aiClient } = await getLaunchDarklyClients();
14
  // Create the user context for this request
15
  const ctx: LDContext = {
16
    // The context key should be unique to this end user
17
    // You can use any generated value
18
    key: "example-user-key",
19
    kind: "user",
20
    name: userName,
21
  };
22
23
  // Retrieve the AgentControl configuration for this context
24
  const aiConfig = await aiClient.completionConfig(
25
    // The AgentControl config key, which you can copy from the sidebar in the LaunchDarkly UI
26
    "chat-helper-v1",
27
    // The context for this request
28
    ctx,
29
    // A fallback configuration
30
    // For this example, skip execution if the AgentControl config is disabled
31
    {},
32
    // The parameters for this AgentControl config
33
    // These will replace the {{}} placeholders in the LaunchDarkly UI
34
    {
35
        preferences: userPreferences,
36
        // Fetch our mock product data and format it for the completion matching the format we provided to the model
37
        productsAvailable: productsAvailable.map(product => formatProductForCompletion(product)),
38
    }
39
  );
40
41
  // If the AgentControl config is disabled, make sure to handle that appropriately
42
  if (!aiConfig.enabled) {
43
    // Application path to take when the aiConfig is disabled
44
    // For example, you could show a message that the shopping assistant
45
    // is not available and customers should try again later
46
    console.log("AgentControl config is disabled");
47
  }
48
49
  // Mint a tracker for this execution.
50
  const tracker = aiConfig.createTracker();
51
52
  // Track the completion and return the result. trackMetricsOf pairs the
53
  // OpenAI call with getAIMetricsFromResponse from the
54
  // @launchdarkly/server-sdk-ai-openai package, which records duration,
55
  // success/error, and token usage automatically.
56
  const completion = await tracker.trackMetricsOf(
57
    getAIMetricsFromResponse,
58
    async () => await openaiClient.chat.completions.create({
59
      model: aiConfig.model!.name as string,
60
      messages: [
61
        // Add the system and assistant messages from the AgentControl config, as well as the user message
62
        ...aiConfig.messages!,
63
        { role: "user", content: userMessage }
64
      ],
65
    }),
66
  );
67
68
    return `<pre>${completion.choices[0].message.content}</pre>`;
69
};

When you wrap the completion call in tracker.trackMetricsOf, paired with getAIMetricsFromResponse from the @launchdarkly/server-sdk-ai-openai package, you can monitor the performance of your application.

Plugging it in

Now that you’ve set up the AgentControl config and completion, you need to plug this into your app somewhere where you can get a response.

Here’s an example:

1
await completeShoppingAssistant("Sandy", "I'm looking for a fitness tracker", ["premium"])

With these parameters, run your application and you should get a response that looks like this:

Hi Sandy! Based on your preference for premium products, here are the fitness trackers that match your request: - [RaceTracker Pro GPS Elite, 16]

You can play around with the preferences or the query passed in to see different results.

At this point, when you run your application, you receive a response from the model that has relevant results to customized to the input, but the formatting isn’t customer-facing. If you were storing your prompts in code, this would require a code change to format differently. However, because you have it in an AgentControl config, you can update the formatting without having to touch the code or redeploy the application.

Step 6: Update configurations dynamically

Next, let’s adjust the formatting. You can do this without changing application code, or even requiring developer involvement. If any member of your team wants to edit the prompt, adjust the formatting or tone, or make any changes, they can do that directly from the LaunchDarkly UI.

In the LaunchDarkly UI, navigate to the Variations tab of the AgentControl config you created earlier. You’ll edit the existing variation to reflect the formatting you’d prefer for the output. You’ll also update the system messaging to be a bit more friendly and less academic.

First, edit the existing variation and adjust the system message to the following:

You are an e-commerce shopping assistant. You help users find products that match their preferences. Your tone should be friendly and warm, as if you're a friend shopping with the person purchasing the items.

Next, adjust the formatting on the assistant response format message to be more customer-facing, provide the result in more nicely-formatted Markdown, and provide a link to the product in the catalog:

Products are listed in the following format: [Product Name, Product ID, Product Type, {Preference tags}]
The user's preferences are {{preferences}}
The products available for purchase are: {{productsAvailable}}
Return the matched products as a list in the following format with each product on a new line:
##[productName](https://my-store-url/product/{productId})\n
productDescription\n\n
##[productName](https://my-store-url/product/{productId})\n
productDescription\n\n
Address your response to {{ldctx.name}}

With these parameters, run your application again:

1
await completeShoppingAssistant("Sandy", "I'm looking for some running shoes", ["athletic", "casual", "running"])

and you should get a response that looks like this:

Hey Sandy! Let's find the perfect running shoes for you. Based on your preferences, here are some options I think you'll love:
##[AeroGlide Elite Runner 3.0](https://my-store-url/product/1)
These shoes are designed for athletic performance and running. They offer incredible comfort, making them a great choice for your runs. Perfect for keeping a steady pace while feeling great on your feet!
##[PowerBoost Pro Runner X22](https://my-store-url/product/5)
These premium running shoes elevate your run with their top-notch design and superior performance. Ideal for serious runners who want that extra boost on the track!
##[CloudStep Comfort Runner](https://my-store-url/product/9)
Focused on comfort, these running shoes are neutral yet supportive, ensuring you have a smooth, enjoyable run every time. They're perfect for everyday runs and long distances alike.
##[UltraGlide Distance Elite](https://my-store-url/product/12)
For those long-distance runs, these premium shoes offer unbeatable performance. They're a great choice if you're looking to cover miles with ease and efficiency.
I hope one of these pairs catches your eye! Let me know if you need more information or help deciding. Happy running!

The output now contains a friendlier message, links to the products, and provides the descriptions in Markdown format. As long as you’re providing the data in the completion input context, you can update this output to contain any of the input parameters without the need for code changes.

Bonus: Track LLM metrics

When you set up the OpenAI completion call, you minted a tracker via aiConfig.createTracker() and passed the OpenAI call to tracker.trackMetricsOf with getAIMetricsFromResponse as the metrics extractor. This pattern automatically captures duration, success/error, and token usage metrics for OpenAI:

1
  const tracker = aiConfig.createTracker();
2
  const completion = await tracker.trackMetricsOf(
3
    getAIMetricsFromResponse,
4
    async () => await openaiClient.chat.completions.create({
5
      model: aiConfig.model!.name as string,
6
      messages: [
7
        // Add the system and assistant messages from the AgentControl config, as well as the user message
8
        ...aiConfig.messages!,
9
        { role: "user", content: userMessage }
10
      ],
11
    }),
12
  );

After you’ve run your completion a few times, check the Monitoring tab for your AgentControl config in the LaunchDarkly UI. The Monitoring tab displays metrics that are automatically tracked, including:

• Generation count
• Input tokens used
• Output tokens used

In combination with LaunchDarkly’s targeting rules, you can duplicate the prompts with different models and messages to see the differences in their generation patterns.

Tracking success

You can also use the LaunchDarkly AI client to keep a running total of positive and negative sentiment about the prompt generation. Feedback must be recorded on the same tracker, with the same runId, as the AI call it is about, or the feedback won’t be attributed to that response. There are two cases to handle: same-request feedback and deferred feedback.

If the feedback signal is available during the same request that produced the response (for example, an automated validation step runs immediately after the completion), reuse the tracker you already minted for trackMetricsOf:

1
// `tracker` here is the same instance you passed to `tracker.trackMetricsOf`
2
// earlier in the request. Do not call `aiConfig.createTracker()` again. If you do,
3
// that creates a fresh tracker with a new `runId`, which causes the feedback to
4
// not be attributed to the original AI call.
5
tracker.trackFeedback({ kind: LDFeedbackKind.Positive | LDFeedbackKind.Negative })

If the feedback arrives later (for example, the user clicks thumbs up or down minutes or hours after the response was shown), the original tracker object is gone. To prevent this, use the resumption-token pattern: at generation time, serialize tracker.resumptionToken alongside the response so your frontend can echo it back. In the feedback handler, call aiClient.createTracker(resumptionToken, ctx) to rebuild a tracker bound to the original runId, and then record feedback on the rebuilt tracker.

1
// Generation time inside completeShoppingAssistant, alongside trackMetricsOf:
2
const resumptionToken = tracker.resumptionToken;
3
// Persist `resumptionToken` with the response so the feedback handler can
4
// retrieve it. For example, return it to the client, or store it keyed by message ID.
5
6
// Feedback time for a different request, possibly a different process:
7
const feedbackTracker = aiClient.createTracker(resumptionToken, ctx);
8
feedbackTracker.trackFeedback({ kind: LDFeedbackKind.Positive })

Conclusion

In this guide, you reviewed how to manage AI model configuration and prompts for an OpenAI-powered application, and how to dynamically customize your application.

Using AgentControl with the LaunchDarkly AI SDKs means you can:

• modify AI prompts and model parameters directly in LaunchDarkly
• empower non-engineers to refine AI behavior without code changes
• gain insights into model performance and token consumption

To learn more, read AgentControl and AI SDKs.