Optimize an AI agent to sound human, judged by an AI detector

Published June 2026

by Scarlett Attensil

Introduction

You can tell when an LLM wrote an email. The “I hope this email finds you well” opener, the three polite paragraphs answering a one-line question. I wanted a reply-drafting agent that didn’t do that, and “don’t sound like an AI” turned out to be hard to put in a prompt. Banning a few phrases is easy. The rest is judgment, and a single prompt that holds across a friendly dinner invite and a recruiter cold-email took more iterations than I’d guessed.

This is not only an email problem. Some platforms down-rank content that reads as AI-generated, so teams publishing at scale have a real stake in prose that clears a detector, even when a human wrote it. The workflow here applies to any of that.

So I stopped hand-tuning and let LaunchDarkly agent optimization search for the prompt. You give it a judge that scores “better,” and it generates prompt variations and keeps the ones that beat the bar. For the reasoning behind the feature, read the agent optimization announcement. This tutorial is the how. If you don’t have an account yet, sign up for LaunchDarkly to follow along.

Two pieces do the work here. Claude (claude-haiku-4-5-20251001) runs both roles: it drafts the replies, and it writes each new candidate prompt when the loop asks for one. Scoring comes from GPTZero, which isn’t a language model at all but a closed AI detector. I wired it in inverted, so the score is the probability a reply reads as AI and the optimizer drives it down. I went with a detector instead of an LLM-as-a-judge for a reason: grading one model’s prose by asking another model whether it sounds human is exactly the call language models are unreliable at, and a tool trained for that one question gives a number you can defend.

A run is cheap. Each iteration costs around $0.002 and a few seconds, so a full run lands near a penny or two, and the loop tries variations I’d never sit down and type by hand.

What you will build

• A reply-drafting agent, email-agent, seeded with one deliberately thin instruction
• An inverted AI-likeness judge, ai-likeness, scored by GPTZero in code rather than by a prompt
• A saved optimization, email-agent-opt, that runs as a candidate generator and streams to the Results tab
• A path from the candidates it surfaces into a fuller offline eval, where the real measurement happens

The GPTZero integration is the reusable part. The same shape works for any external scorer you might bring, whether a moderation API, a classifier you host, or a scoring endpoint of your own, so what you learn here isn’t limited to email or to AI detection.

The companion repo is agent-optimization-sample. Clone it to follow along.

Prerequisites

• Sign up for LaunchDarkly and request access to agent optimization
• Python 3.11+ and uv
• An Anthropic API key, which Claude uses to both draft replies and write each new prompt
• A GPTZero API key for the AI-likeness detector
• A LaunchDarkly SDK key and a REST API key, in a local .env

Install the project and its dependencies:

$
uv sync
$
# .env holds LD_SDK_KEY, LD_API_KEY, LD_PROJECT_KEY,
$
#            ANTHROPIC_API_KEY, AI_LIKENESS_API_KEY

The repo aliases the short LD_* names to the LAUNCHDARKLY_* names the SDK expects, so the short names in .env are enough.

How agent optimization works

Agent optimization runs an iterative loop against an AgentControl config. It measures your current variation as a baseline, generates candidate variations, and scores each against your acceptance criteria. The loop has a simple shape:

┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│   Define    │────▶│   Explore    │────▶│   Commit    │
│  "Better"   │     │  candidates  │     │  a winner   │
└──────▲──────┘     └──────────────┘     └──────┬──────┘
       │                                        │
       └────────────────────────────────────────┘

Define better. You set acceptance criteria with a judge and a threshold. A judge scores a response on one dimension. You reference a judge saved as an AgentControl config by its key.

Explore candidates. Each iteration drafts against your inputs, scores the result, and writes the next candidate from what the scores tell it. The threshold here is a gate that keeps the loop generating. When a candidate clears it, the optimizer re-runs that same prompt against a few more of your input samples and keeps it only if it passes those too, so a prompt that got lucky on one message doesn’t win. Even then, clearing the gate doesn’t certify that a candidate is good enough to ship. A fuller eval decides that, later.

Commit the winner. The recommended variation shows up in LaunchDarkly, and with autoCommit it publishes back to the agent’s Variations tab so you can read what the optimizer wrote.

You also pick an evaluation mode. Exploratory mode infers quality from the judge alone, which suits open-ended inputs that have no single correct output. Expected Output mode scores against known-correct answers. Replies have no single right answer, so this tutorial stays in Exploratory mode.

How the sample is organized

The companion repo keeps the moving parts in small files, so each piece is easy to find and swap:

• bootstrap.py: seeds the three LaunchDarkly objects, the email-agent config, the ai-likeness judge, and the email-agent-opt optimization. It’s safe to re-run, and it prints links to the configs and the Results tab.
• optimize_from_config.py: the one run command. It reads the saved optimization, runs it, streams each iteration to the Results tab, and prints the tab’s link at the end.
• optimize.py: the two callbacks the run needs. handle_agent_call drafts replies on Claude, and writes the next candidate prompt on Claude too when the SDK asks for one. handle_judge_call scores AI-likeness with GPTZero and hands the optimizer the per-reply detector output.
• detector.py: the GPTZero client. score_with_response returns the number the judge gates on and the full GPTZero JSON.
• messages.py: the synthetic input messages.
• gptzero_test.py: a standalone probe for scoring a draft by hand.
• clients.py and env.py: the LaunchDarkly and Anthropic clients, built once each, and the .env loader.

The saved optimization holds what you’re optimizing for: the judge, the threshold, the inputs, and the model choices. The code holds how the work happens, drafting on Claude and scoring with GPTZero. You edit the what in the UI and the how in code, and the run command brings the two together.

Step 1: Bootstrap the agent, judge, and optimization

One command seeds everything this tutorial needs. bootstrap.py creates three objects in LaunchDarkly, and it’s idempotent, so anything that already exists is left alone:

• email-agent: the agent config whose instructions the optimizer tunes. The baseline gives the model the task and the output contract, a JSON {"replies": [...]} envelope and the {{messages}} variable, and nothing about tone. That’s on purpose. It leaves the humanization, the part you want optimized, to the optimizer.
• ai-likeness: the inverted judge config. GPTZero scores it from code, which leaves the judge prompt as a placeholder.
• email-agent-opt: the saved optimization the Results tab runs. Thresholds, inputs, and model choices all live here.

The baseline is thin on tone but carries the output contract the parser needs, on the Claude model the agent drafts with:

1
{
2
    "key": "baseline",
3
    "name": "Baseline",
4
    "model": {"modelName": "claude-haiku-4-5-20251001", "parameters": {}},
5
    "instructions": (
6
        "You are an email assistant. Write a reply to each message below. "
7
        'Return ONLY a JSON object {"replies": ["<reply to Message 1>", ...]} '
8
        "with one reply per message, in order.\n\n{{messages}}"
9
    ),
10
}

Run the bootstrap:

$
uv run python bootstrap.py

It prints links to the two configs and the one command that runs the optimization.

Step 2: Define the input messages

The agent drafts against a fixed set of messages, injected into the prompt through the {{messages}} variable. Keep them diverse on purpose, so a winning prompt generalizes instead of overfitting to one kind of message. The agent replies to all of them in one call and the judge averages the AI-likeness scores, and Step 3 explains why that averaging matters. messages.py holds the message list:

1
SYNTHETIC_MESSAGES = [
2
    "Hey! Are you free Saturday for dinner? A few of us are getting together and it'd be great to see you.",
3
    "Hi, I came across your background and I'm hiring for a senior role that looks like a strong fit. Open to a quick call this week?",
4
    "It's been way too long! I'll be in town Thursday and Friday. Any chance you're around to grab coffee?",
5
    "Just a reminder that your dentist appointment is Tuesday at 2pm. Reply to confirm or reschedule.",
6
    "Hey neighbor, a package addressed to you was left at my door by mistake. Want to swing by this weekend?",
7
    # 5 more, spanning tone, intent, and length
8
]

Step 3: Score AI-likeness with GPTZero

AI-likeness is the target, and it’s the one judge that has to live in code, because it isn’t an LLM. You score the reply with GPTZero, an AI detector. The score is 1 - P(human): near 0 when GPTZero reads the reply as human, which is your goal, and near 1 for AI or mixed text. Lower is better, so the bootstrap marks the judge inverted and the optimizer drives the score down.

GPTZero is also the more useful integration to learn, because it generalizes to any external scorer. The SDK gives a judge config no hook to reach outside code, so the config exists only so the optimization can attach a judge, its prompt stays a placeholder, and the real number comes from a small client you call yourself:

1
def score_with_response(text: str):
2
    data = _api_call(text)                                    # POST the reply to GPTZero
3
    doc = (data.get("documents") or [{}])[0]
4
    return 1.0 - doc["class_probabilities"]["human"], data    # 1 - P(human), + full JSON

score_with_response returns both the number the judge gates on and the full GPTZero JSON, so the callback can forward the detail to the optimizer. Set AI_LIKENESS_API_KEY to your GPTZero key.

Two things here cost me time. GPTZero sits behind Cloudflare, and a plain urllib request comes back as a 403 with error code: 1010 before it reaches the API. That reads like a bad key, but it isn’t. Sending any real User-Agent header clears it. The score also comes from 1 - P(human) rather than the average_generated_prob field, which is the fraction of sentences flagged AI and reports 1.0 even on text the detector still classifies as human. Gating on that field punishes replies that already passed.

GPTZero is the only judge, so handle_judge_call doesn’t branch on the judge key. Every call pulls the batch of replies, scores each one with GPTZero, and averages:

1
async def handle_judge_call(key, config, context, is_evaluation=True):
2
    text = _extract_candidate(context.user_input or "")
3
    replies = [r for r in (json.loads(text) or {}).get("replies", [])
4
               if isinstance(r, str) and r.strip()]
5
    if not replies:                              # empty draft must FAIL the inverted gate
6
        return OptimizationResponse(output=json.dumps(
7
            {"score": 1.0, "rationale": "empty or degenerate candidate"}))
8
    results = []
9
    for reply in replies:                        # one GPTZero call per reply in the batch
10
        s, raw = detector.score_with_response(reply)        # 1 - P(human); + full JSON
11
        results.append({"reply": reply, "ai_likeness": s, "gptzero": raw})
12
    avg = round(sum(r["ai_likeness"] for r in results) / len(results), 4)
13
    # The gate is the AVERAGE; the full per-reply GPTZero JSON becomes the rationale.
14
    return OptimizationResponse(output=json.dumps(
15
        {"score": avg, "rationale": f"Average AI-likeness = {avg}. {json.dumps(results)}"}))

Two choices in there are what make optimizing against a detector actually work.

Score a batch and average. A single short reply’s GPTZero score is noisy. The very same prompt can produce a reply it calls 99% human on one message and 99% AI on the next. Each turn drafts replies to all the messages in one batch, and the judge averages those scores into something steady enough to optimize against.

Forward the whole detector response. The rationale you return goes straight to the model that writes the next prompt. Instead of a bare number, return the full GPTZero JSON, with its per-sentence probabilities and predicted class. The optimizer reads that directly and revises around whatever scored as AI, with no parsing on your side.

The empty-draft case is the one I got wrong first. An empty reply scores 0.0, which the inverted gate reads as perfectly human, so an early version let the optimizer win by drafting nothing. Now an empty or malformed batch, or a detector error, returns 1.0 and fails the gate.

To build intuition before you run the loop, probe GPTZero on a draft by hand:

$
uv run python gptzero_test.py "your draft reply"

Step 4: Run the optimization

The saved optimization holds the judge, the threshold, the inputs, and the model choices, so the run command takes none of them. The agent drafts on Claude, the optimizer writes each new prompt on Claude too, and the threshold keeps the loop generating rather than picking a winner. Here is the saved optimization:

1
{
2
    "key": "email-agent-opt",
3
    "aiConfigKey": "email-agent",
4
    "maxAttempts": 10,
5
    "judgeModel": "claude-haiku-4-5-20251001",       # required by the API; the GPTZero judge never calls it
6
    "modelChoices": ["claude-haiku-4-5-20251001"],   # the Claude model the agent drafts with
7
    "judges": [{"key": "ai-likeness", "threshold": 0.5}],   # generator gate, not a winner test
8
    "variableChoices": [   # interpolated into the instructions; the optimizer must use every one
9
        {"sender_type": "friend", "respondent_name": "Jordan Lee", "messages": MESSAGES_BLOCK},
10
        {"sender_type": "professional contact", "respondent_name": "Jordan Lee", "messages": MESSAGES_BLOCK},
11
    ],
12
    "userInputOptions": ["Draft the replies now."],  # trigger turn; messages come from {{messages}}
13
    "autoCommit": True,
14
}

MESSAGES_BLOCK is the message list from Step 2, formatted and fed in through {{messages}}. respondent_name and sender_type are the other two variables, so replies come out signed and pitched to the right register. The optimizer has to use every variable you declare, which is what keeps {{messages}} and the JSON envelope intact through every rewrite.

The threshold is 0.5, and that number came from watching GPTZero, not from theory. A confident detector scores even clearly human-sounding short replies well above 0, so a gate near 0 never trips and the loop never finds anything to keep. At 0.5, this run passed at iteration 6 with 0.43, while the earlier iterations landed between 0.54 and 1.00. That gave the loop room to explore without rubber-stamping every candidate.

Run it from the saved config:

$
OPTIMIZATION_KEY=email-agent-opt uv run python optimize_from_config.py

The command prints a link to the Results tab. One callback drafts replies on Claude, and the SDK reuses that same callback to write the next prompt, also on Claude. Each iteration posts its prompt and score to the Results tab as a candidate.

Step 5: Read the winner

Open the Results tab from the printed link. Each iteration posts as it runs, with its candidate prompt and AI-likeness average alongside the variation the run currently recommends.

Click any iteration to drill into its candidate prompt, the input it ran against, and the replies it produced. Iteration 1 is the baseline template itself, scoring 0.64. Its replies already read decently (“Thanks for the invite! I’d love to come to dinner Saturday.”), but the thin prompt left enough AI signal for the detector to flag.

The optimization sets autoCommit, so on success the winner publishes back to email-agent as a new variation. Open the agent Variations tab to read what the optimizer wrote.

What the optimizer changed

The run committed a new variation, optimistic-coyote. The Variations tab shows it next to the baseline, so you can read the change directly. The optimizer kept the JSON envelope and the {{sender_type}} and {{messages}} variables, and built a full humanization spec around them:

Every line of it is a humanization lever, and they map onto how GPTZero separates the two classes:

• How people actually write. Contractions, first person, sharply varied sentence length, the occasional fragment, and opening with the point instead of a preamble.
• A stop-list of AI tells. “I hope this email finds you well,” “Thank you for reaching out,” and “Best regards,” plus connectors like “furthermore,” “moreover,” and “consequently.”
• Specific and signed. Match the sender’s register, reference concrete details from the message, and sign with {{respondent_name}} instead of a placeholder.

The optimizer took a prompt that said nothing about tone and built out a detailed spec, and GPTZero scored the resulting replies as more human.

That’s one of the candidates the loop surfaced. To decide whether it’s worth shipping, take it into a fuller offline eval for real data, which the sections below cover.

The UI, the saved config, and the SDK

You can run all of this from the UI or from code. The New optimization form builds the same optimization by hand and streams to the same Results tab. This tutorial used optimize_from_config, which runs a saved optimization from code while its judge, threshold, inputs, and models stay editable in the UI. To define everything in code instead, optimize_from_options takes the settings directly and accepts more than one judge, and optimize_from_ground_truth_options handles Expected Output mode when you have correct answers to match.

This demo leaves several controls unused: token_optimization and latency_optimization for a cost-and-latency pass, token_limit for a spend cap, variation_key, output_key, context_choices, and the on_turn, on_passing_result, on_failing_result, and on_status_update callbacks.

Optimization is not evaluation

Offline evals and optimization sit next to each other in AgentControl and do opposite jobs. An offline eval measures a configuration you already have: you run your agent over a dataset, score it with judges, and answer “how good is this, and did anything regress?” Optimization runs the other direction, generating new variations until one clears that same judge. Running a surfaced candidate back through an eval is that measuring job again, telling you how it actually performs on a fuller set.

They’re strongest together. Here’s a path through AgentControl that gets the most out of both:

1. Define judges for what “better” means, the way you defined human-sounding replies here.
2. Baseline with offline evals over a dataset, so you know where the current agent stands and have a regression check to compare against. For a worked example, read Offline evaluation of RAG-grounded answers.
3. Optimize against that judge to discover a stronger variation, which is what this tutorial walks through.
4. Watch production with online evals and AI Insights, then feed what you learn back into the dataset and the next optimization. When to add online evals covers the tradeoffs.

Run it long enough and production signals become the next round’s eval data, so each optimization starts from what you actually saw in production rather than a guess.

Wrap up

You started with a prompt that said nothing about tone and let agent optimization rewrite it against GPTZero, then read the winning humanization spec off the Variations tab. The loop’s job was to explore cheaply, and the trustworthy verdict comes from a proper offline eval over a comprehensive dataset.

Agent optimization is one step in a larger workflow. You define judges for what “better” means, optimize against them to surface candidates, then check those candidates with offline and online evals. What you learn in production feeds the next round. If you’re getting started, Build a LangGraph multi-agent system is a good place to begin.

Sign up for LaunchDarkly and point the loop at your own prompts.