Managing AI model configuration outside of code with the Node.js AI SDK

This guide shows how to manage AI model configuration and prompts for an OpenAI-powered application in a runtime environment. It uses the LaunchDarkly Node.js (server-side) AI SDK and AgentControl to dynamically customize your application. Using AgentControl to customize your application means you can:

manage your model configuration and prompts outside of your application code
enable non-engineers to iterate on prompt and model configurations
apply updates to prompts and configurations without redeploying your application

This guide steps you through the process of working in your application and in LaunchDarkly to customize your application. If you’re already familiar with setting up AgentControl in the LaunchDarkly UI and want to skip straight to the sample code, you can find it on GitHub.

Additional resources for AgentControlIf you’re not familiar with AgentControl and would like additional explanation, you can start with the Quickstart for AgentControl and come back to this guide when you’re ready for a more realistic example.You can find reference guides for each of the AI SDKs at AI SDKs.

Prerequisites

To complete this guide, you must have the following prerequisites:

a LaunchDarkly account, including
- a LaunchDarkly SDK key for your environment.
- a role that allows AgentControl actions. The LaunchDarkly Project Admin, Maintainer, and Developer project roles, as well as the Admin and Owner base roles, all include this ability.
a Node.js (server-side) application. This guide provides sample code in TypeScript. You can omit the types if you are using JavaScript.
an OpenAI API key. The LaunchDarkly AI SDKs provide specific functions for completions for several common AI model families, and an option to record this information yourself. This guide uses OpenAI.

Example scenario

In this example, you manage a product recommendation system for an e-commerce platform. Using the LaunchDarkly Node.js (server-side) AI SDK, you’ll configure AI prompts to provide personalized product suggestions based on your customers’ preferences. You’ll also track metrics, such as the number of output tokens used by your generative AI application.

Step 1: Prepare your development environment

First, install the required SDKs:

npm install @launchdarkly/node-server-sdk "@launchdarkly/server-sdk-ai@^0.20.0" "@launchdarkly/server-sdk-ai-openai@^0.6.0"

yarn add @launchdarkly/node-server-sdk "@launchdarkly/server-sdk-ai@^0.20.0" "@launchdarkly/server-sdk-ai-openai@^0.6.0"

Then, set up credentials in your environment:

export OPENAI_API_KEY=your_openai_api_key
export LAUNCHDARKLY_SDK_KEY=your_LaunchDarkly_SDK_KEY

If you are starting from scratch, you can use the Node dotenv package in conjunction with a .env file to load your API keys.

Step 2: Initialize LaunchDarkly SDK clients

Inside of your project, create a shared utility for initializing the LaunchDarkly SDK and its AI client. Create a config/launchDarkly.ts file:

import { init, LDClient } from "@launchdarkly/node-server-sdk";
import { initAi, LDAIClient } from "@launchdarkly/server-sdk-ai";

const LD_SDK_KEY = process.env.LAUNCHDARKLY_SDK_KEY;

const ldClient: LDClient = init(LD_SDK_KEY as string);

const aiClient: LDAIClient = initAi(ldClient);
// Initialize and return the LaunchDarkly client
export const getLaunchDarklyClients = async () => {
  try {
    await ldClient.waitForInitialization({ timeout: 10 });
  } catch (err) {
    // log your error
  }
  return { ldClient, aiClient };
};

Step 3: Set up AgentControl configs in LaunchDarkly

Next, create an AgentControl config in the LaunchDarkly UI. AgentControl configs are the LaunchDarkly resources that manage model configurations and messages for your generative AI applications. To create an AgentControl config:

In LaunchDarkly, click Create and choose Config.
In the “Create config” dialog, enter a human-readable Name for your AgentControl config, for example, “chat-helper-v1” or “shopping assistant.”
Click Create.

Then, configure the model and prompt by creating a variation. Every AgentControl config has one or more variations, each of which includes your AI messages and model configuration. Here’s how:

In the create panel in the Variations tab, replace “Untitled variation” with a variation Name. You’ll use this to refer to the variation when you set up targeting rules, below.
Click Select a model and select a supported OpenAI model, for example, gpt-4o.
Optionally, adjust the model parameters: click Parameters to view and update model parameters. In the dialog, adjust the model parameters as needed. The Base value of each parameter is from the model settings. You can choose different values for this variation if you prefer.

Next, add system, user, or assistant messages to define your prompt. In this example, you’ll augment the prompt with saved user preferences. Start with the system message:

You are an e-commerce shopping assistant. You help users find products that match their preferences.

Now, add an assistant message that pre-loads the model with the user’s preferences and provides some instructions about the input format, response format, and context:

Products are listed in the following format: [Product Name, Product ID, Product Type, {Preference tags}]

The user's preferences are {{preferences}}

The products available for purchase are: {{productsAvailable}}

Return the matched products as a list in the following format:
- [productName, productId]
- [productName, productId]

Address the response to {{ldctx.name}}

The double curly braces in the prompts allow you to augment the messages with customized data at runtime. To learn more, read Syntax for customization. Within the UI, the variation will look like the following:

The completed variation for your AgentControl config.

Click Review and save at the top right of the page to save the configuration.

Step 4: Enable targeting in LaunchDarkly

Targeting is how you specify that specific end users working in your application will receive specific variations that you’ve created in your AgentControl config. To set up targeting, click the Targeting tab. Targeting for AgentControl configs is on by default. Set the default targeting rule to serve the “Shopping preference assistant” variation you just created for your test environment. To specify the AgentControl config variation to use by default when the AgentControl config is toggled on:

Select the Targeting tab for your AgentControl config.
In the “Default rule” section, click Edit.
Set the default rule to serve the shopping preference assistant variation.
Click Review and save. In the confirmation dialog, click Save changes.

By default, this AgentControl config will now serve the variation shopping preference assistant variation:

The default rule in your AgentControl config.

Step 5: Integrate the AgentControl config into your application

To integrate the LaunchDarkly AgentControl config into your application, you’ll need to set up your OpenAI client, set up your application data, add your completion code, and finally put it all together.

Set up the OpenAI client

Next, set up your OpenAI client. Following a similar pattern as above, create a config/openAi.ts file to initialize the OpenAI client:

import { OpenAI } from "openai";

let openaiClient: OpenAI;

export const getOpenAI = () => {
    if(!openaiClient) {
        openaiClient = new OpenAI();
    }
    return openaiClient;
}

Set up application data

Then, set up the data for your application. Normally, the product data that your application retrieves would come from a database. For this example, create a data/products.ts file to serve mock data:

import { productsAvailable } from "./mockProducts";

export type Product = {
    name: string;
    productId: string;
    description: string;
    price: number;
    type: "shoes" | "clothing" | "accessories";
    preferenceTags: string[];
}

export const getProducts = () => {
    return {
        productsAvailable,
    }
}

// format is [Product Name, Product ID, Product Type, {Preference tags}]
export const formatProductForCompletion = (product: Product) => {
    return `[${product.name}, ${product.productId}, ${product.type}, ${product.preferenceTags.join(", ")}]`
}

Additionally, add the following data/mockProducts.ts file in the same folder:

import { Product} from "./products"

export const productsAvailable: Product[] = [
    {
        name: "AeroGlide Elite Runner 3.0",
        productId: "1",
        description: "Versatile running shoes with responsive cushioning",
        type: "shoes",
        price: 120,
        preferenceTags: ["athletic", "running", "comfort"]
    },
    // complete list of products available in linked file
  ]

The data contains the name, price, type, preference tags to match to a customer’s preferences, a description, and a product ID. This should give the AI model ample data to match to a customer’s query.

Add your completion

Next, add the AgentControl config and completion code into your application. Create a completions/shoppingAssistant.ts file:

import { LDContext } from "@launchdarkly/node-server-sdk";
import { getLaunchDarklyClients } from "../config/launchdarkly";
import { getOpenAI } from "../config/openAI";
import { formatProductForCompletion, getProducts } from "../data/products";
import { LDFeedbackKind } from "@launchdarkly/server-sdk-ai";
import { getAIMetricsFromResponse } from "@launchdarkly/server-sdk-ai-openai";

export const completeShoppingAssistant = async (userName: string, userMessage: string, userPreferences: string[]) => {
  const openaiClient = getOpenAI();

  const { productsAvailable } = await getProducts();

  const { aiClient } = await getLaunchDarklyClients();
  // Create the user context for this request
  const ctx: LDContext = {
    // The context key should be unique to this end user
    // You can use any generated value
    key: "example-user-key",
    kind: "user",
    name: userName,
  };

  // Retrieve the AgentControl configuration for this context
  const aiConfig = await aiClient.completionConfig(
    // The AgentControl config key, which you can copy from the sidebar in the LaunchDarkly UI
    "chat-helper-v1",
    // The context for this request
    ctx,
    // A fallback configuration
    // For this example, skip execution if the AgentControl config is disabled
    {},
    // The parameters for this AgentControl config
    // These will replace the {{}} placeholders in the LaunchDarkly UI
    {
        preferences: userPreferences,
        // Fetch our mock product data and format it for the completion matching the format we provided to the model
        productsAvailable: productsAvailable.map(product => formatProductForCompletion(product)),
    }
  );

  // If the AgentControl config is disabled, make sure to handle that appropriately
  if (!aiConfig.enabled) {
    // Application path to take when the aiConfig is disabled
    // For example, you could show a message that the shopping assistant
    // is not available and customers should try again later
    console.log("AgentControl config is disabled");
  }

  // Mint a tracker for this execution.
  const tracker = aiConfig.createTracker();

  // Track the completion and return the result. trackMetricsOf pairs the
  // OpenAI call with getAIMetricsFromResponse from the
  // @launchdarkly/server-sdk-ai-openai package, which records duration,
  // success/error, and token usage automatically.
  const completion = await tracker.trackMetricsOf(
    getAIMetricsFromResponse,
    async () => await openaiClient.chat.completions.create({
      model: aiConfig.model!.name as string,
      messages: [
        // Add the system and assistant messages from the AgentControl config, as well as the user message
        ...aiConfig.messages!,
        { role: "user", content: userMessage }
      ],
    }),
  );

    return `<pre>${completion.choices[0].message.content}</pre>`;
};

When you wrap the completion call in tracker.trackMetricsOf, paired with getAIMetricsFromResponse from the @launchdarkly/server-sdk-ai-openai package, you can monitor the performance of your application.

Plugging it in

Now that you’ve set up the AgentControl config and completion, you need to plug this into your app somewhere where you can get a response. Here’s an example:

await completeShoppingAssistant("Sandy", "I'm looking for a fitness tracker", ["premium"])

With these parameters, run your application and you should get a response that looks like this:

Hi Sandy! Based on your preference for premium products, here are the fitness trackers that match your request: - [RaceTracker Pro GPS Elite, 16]

You can play around with the preferences or the query passed in to see different results. At this point, when you run your application, you receive a response from the model that has relevant results to customized to the input, but the formatting isn’t customer-facing. If you were storing your prompts in code, this would require a code change to format differently. However, because you have it in an AgentControl config, you can update the formatting without having to touch the code or redeploy the application.

Step 6: Update configurations dynamically

Next, let’s adjust the formatting. You can do this without changing application code, or even requiring developer involvement. If any member of your team wants to edit the prompt, adjust the formatting or tone, or make any changes, they can do that directly from the LaunchDarkly UI. In the LaunchDarkly UI, navigate to the Variations tab of the AgentControl config you created earlier. You’ll edit the existing variation to reflect the formatting you’d prefer for the output. You’ll also update the system messaging to be a bit more friendly and less academic. First, edit the existing variation and adjust the system message to the following:

You are an e-commerce shopping assistant. You help users find products that match their preferences. Your tone should be friendly and warm, as if you're a friend shopping with the person purchasing the items.

Next, adjust the formatting on the assistant response format message to be more customer-facing, provide the result in more nicely-formatted Markdown, and provide a link to the product in the catalog:

Products are listed in the following format: [Product Name, Product ID, Product Type, {Preference tags}]

The user's preferences are {{preferences}}

The products available for purchase are: {{productsAvailable}}

Return the matched products as a list in the following format with each product on a new line:
##[productName](https://my-store-url/product/{productId})\n
productDescription\n\n
##[productName](https://my-store-url/product/{productId})\n
productDescription\n\n

Address your response to {{ldctx.name}}

With these parameters, run your application again:

await completeShoppingAssistant("Sandy", "I'm looking for some running shoes", ["athletic", "casual", "running"])

and you should get a response that looks like this:

Hey Sandy! Let's find the perfect running shoes for you. Based on your preferences, here are some options I think you'll love:
##[AeroGlide Elite Runner 3.0](https://my-store-url/product/1)
These shoes are designed for athletic performance and running. They offer incredible comfort, making them a great choice for your runs. Perfect for keeping a steady pace while feeling great on your feet!
##[PowerBoost Pro Runner X22](https://my-store-url/product/5)
These premium running shoes elevate your run with their top-notch design and superior performance. Ideal for serious runners who want that extra boost on the track!
##[CloudStep Comfort Runner](https://my-store-url/product/9)
Focused on comfort, these running shoes are neutral yet supportive, ensuring you have a smooth, enjoyable run every time. They're perfect for everyday runs and long distances alike.
##[UltraGlide Distance Elite](https://my-store-url/product/12)
For those long-distance runs, these premium shoes offer unbeatable performance. They're a great choice if you're looking to cover miles with ease and efficiency.
I hope one of these pairs catches your eye! Let me know if you need more information or help deciding. Happy running!

The output now contains a friendlier message, links to the products, and provides the descriptions in Markdown format. As long as you’re providing the data in the completion input context, you can update this output to contain any of the input parameters without the need for code changes.

Bonus: Track LLM metrics

When you set up the OpenAI completion call, you minted a tracker via aiConfig.createTracker() and passed the OpenAI call to tracker.trackMetricsOf with getAIMetricsFromResponse as the metrics extractor. This pattern automatically captures duration, success/error, and token usage metrics for OpenAI:

  const tracker = aiConfig.createTracker();
  const completion = await tracker.trackMetricsOf(
    getAIMetricsFromResponse,
    async () => await openaiClient.chat.completions.create({
      model: aiConfig.model!.name as string,
      messages: [
        // Add the system and assistant messages from the AgentControl config, as well as the user message
        ...aiConfig.messages!,
        { role: "user", content: userMessage }
      ],
    }),
  );

After you’ve run your completion a few times, check the Monitoring tab for your AgentControl config in the LaunchDarkly UI. The Monitoring tab displays metrics that are automatically tracked, including:

Generation count
Input tokens used
Output tokens used

In combination with LaunchDarkly’s targeting rules, you can duplicate the prompts with different models and messages to see the differences in their generation patterns.

Tracking success

You can also use the LaunchDarkly AI client to keep a running total of positive and negative sentiment about the prompt generation. Feedback must be recorded on the same tracker, with the same runId, as the AI call it is about, or the feedback won’t be attributed to that response. There are two cases to handle: same-request feedback and deferred feedback. If the feedback signal is available during the same request that produced the response (for example, an automated validation step runs immediately after the completion), reuse the tracker you already minted for trackMetricsOf:

// `tracker` here is the same instance you passed to `tracker.trackMetricsOf`
// earlier in the request. Do not call `aiConfig.createTracker()` again. If you do,
// that creates a fresh tracker with a new `runId`, which causes the feedback to
// not be attributed to the original AI call.
tracker.trackFeedback({ kind: LDFeedbackKind.Positive | LDFeedbackKind.Negative })

If the feedback arrives later (for example, the user clicks thumbs up or down minutes or hours after the response was shown), the original tracker object is gone. To prevent this, use the resumption-token pattern: at generation time, serialize tracker.resumptionToken alongside the response so your frontend can echo it back. In the feedback handler, call aiClient.createTracker(resumptionToken, ctx) to rebuild a tracker bound to the original runId, and then record feedback on the rebuilt tracker.

// Generation time inside completeShoppingAssistant, alongside trackMetricsOf:
const resumptionToken = tracker.resumptionToken;
// Persist `resumptionToken` with the response so the feedback handler can
// retrieve it. For example, return it to the client, or store it keyed by message ID.

// Feedback time for a different request, possibly a different process:
const feedbackTracker = aiClient.createTracker(resumptionToken, ctx);
feedbackTracker.trackFeedback({ kind: LDFeedbackKind.Positive })

Conclusion

In this guide, you reviewed how to manage AI model configuration and prompts for an OpenAI-powered application, and how to dynamically customize your application. Using AgentControl with the LaunchDarkly AI SDKs means you can:

modify AI prompts and model parameters directly in LaunchDarkly
empower non-engineers to refine AI behavior without code changes
gain insights into model performance and token consumption

To learn more, read AgentControl and AI SDKs.

Want to know more? Start a trial.Your 14-day trial begins as soon as you sign up. Get started in minutes using the in-app Quickstart. You’ll discover how easy it is to release, monitor, and optimize your software.

Want to try it out? Start a trial.

​Prerequisites

​Example scenario

​Step 1: Prepare your development environment

​Step 2: Initialize LaunchDarkly SDK clients

​Step 3: Set up AgentControl configs in LaunchDarkly

​Step 4: Enable targeting in LaunchDarkly

​Step 5: Integrate the AgentControl config into your application

​Set up the OpenAI client

​Set up application data

​Add your completion

​Plugging it in

​Step 6: Update configurations dynamically

​Bonus: Track LLM metrics

​Tracking success

​Conclusion

Prerequisites

Example scenario

Step 1: Prepare your development environment

Step 2: Initialize LaunchDarkly SDK clients

Step 3: Set up AgentControl configs in LaunchDarkly

Step 4: Enable targeting in LaunchDarkly

Step 5: Integrate the AgentControl config into your application

Set up the OpenAI client

Set up application data

Add your completion

Plugging it in

Step 6: Update configurations dynamically

Bonus: Track LLM metrics

Tracking success

Conclusion