Using targeting to manage AI model usage by tier with the Python AI SDK

Overview

This guide shows how to manage AI model usage by customer tier in an OpenAI-powered application. It uses the LaunchDarkly Python AI SDK and AgentControl to dynamically adjust the model used based on customer details.

Using AgentControl and targeting to customize your applications means you can:

• serve different models or messages to different customers, based on attributes of those customers. You can configure this targeting in LaunchDarkly, and update it without redeploying your application.
• compare variations and determine which one performs better, based on satisfaction, cost, or other metrics.

This guide steps you through the process of working in your application and in LaunchDarkly to customize your application and its targeting.

Prerequisites

To complete this guide, you must have the following prerequisites:

• a LaunchDarkly account, including
  • a LaunchDarkly SDK key for your environment.
  • a role that allows AgentControl actions. The LaunchDarkly Project Admin, Maintainer, and Developer project roles, as well as the Admin and Owner base roles, all include this ability.
• a Python development environment. The LaunchDarkly Python AI SDK is compatible with Python 3.8.0 and higher.
• an OpenAI API key. The LaunchDarkly AI SDKs provide specific functions for completions for several common AI model families, and an option to record this information yourself. This guide uses OpenAI.

Example scenario

In this example, you have an application that provides chat support. When creating your generated content, you want to use one AI model for the content you provide to the customers who are paying you, and a different AI model for the content you provide to the customers on your free tier. You also want to understand whether your paying customers are getting a better experience.

Step 1: Prepare your development environment

First, install the Python AI SDK:

$
pip install launchdarkly-server-sdk
$
pip install "launchdarkly-server-sdk-ai>=0.20.0"

Then, set up credentials in your environment. The example below uses $Environment_SDK_KEY and $Environment_OPENAI_KEY to refer to an active LaunchDarkly SDK key and your OpenAI key, respectively.

You can find your SDK keys on the SDK keys page under Settings. To learn more, read SDK credentials.

Step 2: Initialize LaunchDarkly SDK clients

Next, import the LaunchDarkly LDAIClient into your application code and initialize it:

1
import ldclient
2
from ldclient import Context
3
from ldclient.config import Config
4
from ldai import LDAIClient, AICompletionConfigDefault
5
from ldai_openai import get_ai_metrics_from_response
6
7
ldclient.set_config(Config('$Environment_SDK_KEY'))
8
9
if not ldclient.get().is_initialized():
10
    print('SDK failed to initialize')
11
    exit()
12
13
print('SDK successfully initialized')
14
15
aiclient = LDAIClient(ldclient.get())

Step 3: Set up AgentControl configs in LaunchDarkly

Next, create an AgentControl config in the LaunchDarkly UI. AgentControl configs are the LaunchDarkly resources that manage model configurations and messages for your generative AI applications.

To create an AgentControl config:

1. In LaunchDarkly, click Create and choose AgentControl config.
2. In the “Create AgentControl config” dialog, give your AgentControl config a human-readable Name, for example, “Chat bot summarizer.”
3. Click Create.

Then, create two variations. Every AgentControl config has one or more variations, each of which includes your AI messages and model configuration.

Here’s how:

1. On the Variations tab of your newly created AgentControl config, replace “Untitled variation” with a variation Name in the create panel. You’ll use this name to refer to the variation when you set up targeting rules, below. For example, you can use “Premium chat support” for one variation and “Free chat support” for the other variation.
2. Click Select a model and select a supported OpenAI model. For example, you can use “gpt-4o” for your premium variation and “gpt-4-turbo” for your free variation.
3. Optionally, adjust the model parameters: click Parameters to view and update model parameters. In the dialog, adjust the model parameters as needed. The Base value of each parameter is from the model settings. You can choose different values for this variation if you prefer.
4. Add system, user, or assistant messages to define your prompt. For this example, enter a system message for each variation:

You are an expert AI assistant with comprehensive knowledge across multiple domains. Your responses should be detailed, thorough, and professional. You can:
1. Provide in-depth technical explanations
2. Offer multiple solution approaches when applicable
3. Include relevant code examples and best practices
4. Share industry insights and advanced tips
5. Suggest optimizations and improvements
6. Reference technical documentation and trusted sources
Always maintain a professional yet friendly tone, and prioritize accuracy and completeness in your responses.
If a question is unclear, ask for clarification to ensure you provide the most valuable assistance possible.

5. Click Save changes after you create each variation.

Here’s how the two variations should look after you’ve set them up:

Step 4: Set up targeting rules and enable your AgentControl config

Next, set up targeting rules for your AgentControl config. These rules determine which of your customers receives which variation of your AgentControl config.

To specify the AgentControl config variation to use by default when the AgentControl config is toggled on:

1. Select the Targeting tab for your AgentControl config.
2. In the “Default rule” section, click Edit.
3. Configure the default rule to serve the “Free chat support” variation.
4. Click Review and save.

To specify a different AgentControl config variation to use for premium customers:

1. Select the Targeting tab for your AgentControl config.
2. If the AgentControl config is off and the rules are hidden, click View targeting rules.
3. Click the + button between existing rules, and select Build a custom rule.
4. Optionally, enter a name for the rule.
5. Leave the Context kind menu set to “user.”
6. In the Attribute menu, type in customer_type. You’ll set this attribute in your code later.
7. Leave the Operator menu set to “is one of.”
8. In the Values menu, type in premium. You’ll set this value in your code later.
9. From the Select… menu, choose the “Premium chat support” variation.
10. Click Review and save.

By default, the AgentControl config is set to On. Click Review and save.

Here’s what the Targeting tab of your AgentControl config should look like:

Step 5: Customize the AgentControl config

In your code, use the completion_config function in the LaunchDarkly AI SDK to customize the config. You need to call completion_config each time you generate content from your AI model.

The completion_config function returns the customized config, which includes the messages and model configuration for the variation that the end user should receive. Customization is what happens when your application’s code sends the LaunchDarkly AI SDK information about a particular config and the end user that has encountered it in your app, and the SDK sends back the value of the variation that end user should receive.

To call the completion_config function, you need to provide information about the end user who is working in your application. For example, you may have this information in a user profile within your app.

Create a tracker for each generation by calling config.create_tracker(), then wrap your chat completion call with tracker.track_metrics_of. The track_metrics_of method runs the wrapped function, then uses the extractor (get_ai_metrics_from_response) to record duration, token usage, and success or error for that generation.

Here’s how:

1
#... Existing code from Step 2, above
2
3
# OpenAI API Key
4
openai.api_key = "$Environment_OPENAI_KEY"
5
6
# The context describes the end user currently working in your application.
7
# The targeting rules for your AgentControl config can use any context attributes.
8
# This example checks 'customer_type' in one of the targeting rules.
9
context = Context.builder('example-context-key') \
10
    .kind('user') \
11
    .set('name', 'Sandy') \
12
    .set('customer_type', 'premium') \
13
    .build()
14
15
aiclient = LDAIClient(ldclient.get())
16
17
# In case you cannot reach LaunchDarkly
18
fallback_value = AICompletionConfigDefault(enabled=False)
19
20
# Get AgentControl config from LaunchDarkly
21
def get_prompt_and_model():
22
    config = aiclient.completion_config("chat-bot-summarizer", context, fallback_value)
23
    if config.enabled:
24
      return config
25
    else:
26
      # Application path to take when the config is disabled.
27
      # For example, you may want to display a message
28
      # that the chatbot is not available and customers should try again later.
29
30
# Perform a chat completion call
31
def perform_chat():
32
    ai_config = get_prompt_and_model()
33
    tracker = ai_config.create_tracker()
34
35
    # Transform the prompt for OpenAI's format
36
    messages = [{"role": msg.role, "content": msg.content} for msg in ai_config.messages]
37
38
    try:
39
        completion = tracker.track_metrics_of(
40
            get_ai_metrics_from_response,
41
            lambda: openai.chat.completions.create(
42
                model=ai_config.model.name,
43
                messages=messages,
44
            )
45
        )
46
47
    except Exception as e:
48
        print(f"Error during chat completion: {e}")
49
50
if __name__ == "__main__":
51
    perform_chat()

Step 6: Monitor results

As customers encounter chat support in your application, LaunchDarkly monitors the performance of your AgentControl config: the tracker created from the result of the completion_config function automatically records various metrics. To view them, select the Monitoring tab for your config in LaunchDarkly.

In this example, you can review the results to determine:

• which support option provides higher satisfaction for customers
• which support option uses more tokens

You could use this information to make a business decision about whether the performance differences are worth the cost differences of running each model for your different customer tiers.

To learn more, read Monitor AgentControl configs.

Conclusion

In this guide, you learned how to manage AI model usage by customer tier in an OpenAI-powered application, and how to review the performance of those models based on customer feedback and token usage.

For additional examples, read the other AgentControl guides in this section. To learn more, read AgentControl and AI SDKs.