> ## Documentation Index
> Fetch the complete documentation index at: https://launchdarkly.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Day 8 | 🎁 Observable multi-modal agentic systems

export const Carousel = ({items = []}) => {
  const [active, setActive] = useState(0);
  const contentRef = useRef(null);
  const itemRefs = useRef([]);
  useEffect(() => {
    const container = contentRef.current;
    if (!container) return;
    const observer = new IntersectionObserver(entries => {
      entries.forEach(entry => {
        if (entry.isIntersecting) {
          const index = itemRefs.current.indexOf(entry.target);
          if (index !== -1) setActive(index);
        }
      });
    }, {
      root: container,
      threshold: 0.6
    });
    itemRefs.current.forEach(el => el && observer.observe(el));
    return () => observer.disconnect();
  }, [items.length]);
  const scrollTo = index => {
    itemRefs.current[index]?.scrollIntoView({
      behavior: "smooth",
      inline: "center",
      block: "nearest"
    });
  };
  return <div className="relative max-w-full my-8">
      <div ref={contentRef} className="flex overflow-x-auto snap-x snap-mandatory gap-4 py-4 [-webkit-overflow-scrolling:touch]">
        {items.map((item, index) => <div key={item.src} ref={el => itemRefs.current[index] = el} className="shrink-0 grow-0 basis-full snap-center">
            <Frame caption={item.caption}>
              <img src={item.src} alt={item.alt} />
            </Frame>
          </div>)}
      </div>
      <div className="flex justify-center gap-2 mt-4">
        {items.map((_, index) => <span key={index} onClick={() => scrollTo(index)} className={`w-2.5 h-2.5 rounded-full cursor-pointer transition-colors duration-300 ${index === active ? "bg-[#333] dark:bg-white" : "bg-[#ccc] dark:bg-[#555]"}`} />)}
      </div>
    </div>;
};

export const BlogAuthor = ({date, avatar, alt, name}) => {
  return <div className="-mb-4">
      <p className="m-0 text-base font-semibold">
        <em>Published {date}</em>
      </p>
      <div className="flex items-center -mt-4">
        {avatar && <img src={avatar} alt={alt} className="h-8 w-8 rounded-full m-0" />}
        <p className="ml-[5px] text-lg">{name}</p>
      </div>
    </div>;
};

<View title="Developer" />

<View title="Federal docs" />

<View title="EU docs" />

<BlogAuthor date="December 10th, 2025" avatar="/images/authors/scarlettattensil.png" alt="Portrait of Scarlett Attensil." name="by Scarlett Attensil" />

<Check>
  **Newer features are available with AgentControl**

  This tutorial was published in December 2025, before LaunchDarkly shipped several features that extend the observability pattern shown below. The walkthrough still works, but for new builds you may also want to use:

  * [**Custom judges**](/docs/home/agentcontrol/judges): Define multi-modal-specific judges beyond the built-in accuracy/relevance/toxicity set
  * [**Agent graphs**](/docs/home/agentcontrol/agent-graphs): Visualize and monitor per-node latency for multi-modal workflows
  * [**Offline evaluations**](/docs/home/agentcontrol/offline-evaluations) and [**Datasets**](/docs/home/agentcontrol/datasets): Pair production session-replay observability with offline regression scoring
  * [**Manual LLM span tracing**](/docs/home/agentcontrol/manual-llm-span-tracing): Add custom spans for the image-generation and multi-modal handoff steps

  To learn more, read [AgentControl](/docs/home/agentcontrol).
</Check>

I added LaunchDarkly observability to my Christmas-play pet casting app thinking I'd catch bugs. Instead, I unwrapped the perfect gift 🎁. Session replay shows me WHAT users do, and online evaluations show me IF my model made the right casting decision with real-time accuracy scores. Together, they're like milk 🥛 and cookies 🍪 - each good alone, but magical together for production AI monitoring.

## See the App in Action

<Carousel
  items={[
{
  src: "/images/tutorials/observability-multimodal-agents/welcome_screen_pet_casting.png",
  alt: "Welcome Screen",
  caption: "1/6: Welcome Screen - Start your pet's magical journey",
},
{
  src: "/images/tutorials/observability-multimodal-agents/personality_quiz.png",
  alt: "Personality Quiz",
  caption: "2/6: Personality Quiz - Answer fun questions about your pet",
},
{
  src: "/images/tutorials/observability-multimodal-agents/photo_upload.png",
  alt: "Photo Upload",
  caption: "3/6: Photo Upload - Add a photo for personalized casting",
},
{
  src: "/images/tutorials/observability-multimodal-agents/ai_casting_result.png",
  alt: "AI Casting Result",
  caption: "4/6: AI Casting Result - Discover your pet's nativity role",
},
{
  src: "/images/tutorials/observability-multimodal-agents/ai_result_with_evaluation.png",
  alt: "AI Result with Evaluation",
  caption: "5/6: AI Evaluation - See your complete result with score",
},
{
  src: "/images/tutorials/observability-multimodal-agents/another_generation.png",
  alt: "Another Generation",
  caption: "6/6: Another Generation - Generate custom pet costumes",
},
]}
/>

<Note>
  **Tip:** Swipe or scroll horizontally to navigate through the screenshots
</Note>

## Discovery #1: Users' 40-second patience threshold

I decided to use session replay to evaluate the average time it took users to go through each step in the AI casting process. Session replay is LaunchDarkly's tool that records user interactions in your app - every click, hover, and page navigation - so you can watch exactly what users experience in real-time.

The complete AI casting process takes 30-45 seconds: personality analysis (2-3s), role matching (1-2s), DALL-E 3 costume generation (25-35s), and evaluation scoring (2-3s). That's a long time to stare at a loading spinner wondering if something broke.

### What are progress steps?

Progress steps are UI elements I added to the app - not terminal commands or backend processes, but actual visual indicators in the web interface that show users which phase of the AI generation is currently running. These appear as a simple list in the loading screen, updating in real-time as each AI task completes. No commands needed - they automatically display when the user clicks "Get My Role!" and the AI processing begins.

### Session replay revealed:

```expandable lines wrap theme={null}
WITHOUT Progress Steps (n=20 early sessions):
0-10 seconds: 20/20 still watching (100%)
10-20 seconds: 18/20 still watching (90%)
20-30 seconds: 14/20 still watching (70%) - rage clicks begin
30-40 seconds: 9/20 still watching (45%) - tab switching detected
40+ seconds: 7/20 still watching (35% stay)

WITH Progress Steps (n=30 after adding them):
0-10 seconds: 30/30 still watching (100%)
10-20 seconds: 29/30 still watching (97%)
20-30 seconds: 25/30 still watching (83%)
30-40 seconds: 23/30 still watching (77%)
40+ seconds: 24/30 still watching (80% stay!)

Critical Discovery: Progress steps more than DOUBLED
completion rate (35% → 80%)
```

### This made the difference:

**Clear progress steps:**

```lines wrap theme={null}
Step 1: AI Casting Decision
Step 2: Generating Costume Image (10-30s)
Step 3: Evaluation

As each completes:
✅ Step 1: AI Casting Decision
Step 2: Generating Costume Image (10-30s)
Step 3: Evaluation
```

Session replay showed users hovering over the back button at 25 seconds, then relaxing when they saw "Step 2: Generating Costume Image (10-30s)." The moment they understood DALL-E was creating their pet's costume (not the app freezing), they were willing to wait. Clear progress indicators transform anxiety into patience.

## Discovery #2: Observability + online evaluations give the complete picture

Session replay shows user behavior and experience. Online evaluations expose AI output quality through accuracy scoring. Together, they form a solid strategy for AI observability.

To see this in action, let's take a closer look at an example.

### Example: The speed-running corgi owner

In this scenario, a user blazes through the entire pet app setup from the initial quiz to the final results, completing the process in record time. So fast, in fact, that instead of this leading to a favorable outcome, it led to an instance of speed killing quality.

**Session Replay Showed:**

* Quiz completed in 8 seconds (world record) - they clicked the first option for every question
* Skipped photo upload entirely
* Waited the full 31 seconds for processing
* Got their result: "Sheep"
* Started rage clicking on the sheep image immediately
* Left the site without saving or sharing

Why did their energetic corgi get cast as a sheep? The rushed quiz responses created a contradictory personality profile that confused the AI. Without a photo to provide visual context, the model defaulted to its safest, most generic casting choice.

**Online Evaluation Results:**

* Evaluation Score: 38/100 ❌
* Reasoning: "Costume contains unsafe elements: eyeliner, ribbons"
* Wait, what? The AI suggested face paint and ribbons, evaluation said NO

Online evaluations use a model-agnostic evaluation (MAE) - an AI agent that evaluates other AI outputs for quality, safety, or accuracy. The out-of-the-box evaluation judge is overly cautious about physical safety. For the above scenario, the evaluation comments:

* "Costume includes eyeliner which could be harmful to pets" (It's a DALL-E image!)
* "Ribbons pose entanglement risk"
* "Bells are a choking hazard" (It's AI-generated art!)

About 40% of low scores are actually the evaluation being overprotective about imaginary safety issues, not bad casting.

Speed-runners get generic roles AND the evaluation writes safety warnings about digital costumes. Users see these low scores and think the app doesn't work well.

But speed-running isn't the whole story. To truly understand the relationship between user engagement and AI quality, we need to see the flip side. The perfect user. One who gives the AI everything it needed to succeed. What happens when a user takes their time and engages thoughtfully with every step?

### Example: The perfect match

**Session Replay Showed:**

* 45 seconds on quiz (reading each option)
* Uploaded photo, waited for processing
* Spent 2 minutes on results page
* Downloaded image multiple times

**Online Evaluation Results:**

* Evaluation Score: 96/100 ⭐⭐⭐⭐⭐
* Reasoning: "Personality perfectly matches role archetype"
* Photo bonus: "Visual traits enhanced casting accuracy"

Time invested = Quality received. The AI rewards thoughtfulness.

## Discovery #3: The photo upload comedy gold mine

Session replay revealed what photos people ACTUALLY upload. Without it, you'd never know that one in three photo uploads are problematic, and you'd be flying blind on whether to add validation or trust your model.

### Example: The surprising photo upload analysis

**Session Replay Showed:**

```lines wrap theme={null}
Photo Upload Analysis (n=18 who uploaded):
- 12 (67%) Normal pet photos
- 2 (11%) Screenshots of pet photos on their phone
- 1 (6%) Multiple pets in one photo (chaos)
- 1 (6%) Blurry "pet in motion" disaster
- 1 (6%) Stock photo of their breed (cheater!)
```

Despite 33% problematic inputs, evaluation scores remained high (87-91/100). The AI is remarkably resilient.

### Example: When "bad" photos produce great results

**My Favorite Session:** Someone uploaded a photo of their cat mid-yawn. The AI vision model described it as "displaying fierce predatory behavior." The cat was cast as a "Protective Father." Evaluation score: 91/100. The owner downloaded it immediately.

**The Winner:** Someone's hamster photo that was 90% cage bars. The AI somehow extracted "small fuzzy creature behind geometric patterns" and cast it as "Shepherd" because "clearly experienced at navigating barriers." Evaluation score: 87/100.

Without session replay, you'd only see evaluation scores and think "the AI is working well." But session replay reveals users are uploading screenshots and blurry photos—input quality issues that could justify adding photo validation.

However, the high evaluation scores prove the AI handles imperfect real-world data gracefully. This insight saved me from over-engineering photo validation that would have slowed down the user experience for minimal quality gains.

Session replay + online evaluations together answered the question "Should I add photo validation?" The answer: No. Trust the model's resilience and keep the experience frictionless.

## The magic formula: Why this combo works (and what surprised me)

### Without Observability:

* "The app seems slow" → ¯\\*(ツ)*/¯
* "We have 20 visitors but 7 completions" → Where do they drop?

### With Session Replay ONLY:

* "User got sheep and rage clicked; maybe left angry" → Was this a bad match?

### With Model-Agnostic Evaluation ONLY:

* "Evaluation: 22/100 - Eyeliner unsafe for pets" → How did the user react?
* "Evaluation: 96/100 - Perfect match!" → How did this compare to the image they uploaded?

### With BOTH:

* "User rushed, got sheep with ribbons, evaluation panicked about safety"
  → The OOTB evaluation treats image generation prompts like real costume instructions

* "40% of low scores are costume safety, not bad matching"
  → Need custom evaluation criteria (coming soon!)

* "Users might think low score = bad casting, but it's often = protective evaluation"
  → Would benefit from custom evaluation criteria to avoid this confusion

The evaluation thinks we're putting actual ribbons on actual cats. It doesn't realize these are AI-generated images. So when the casting suggests "sparkly collar with bells," the evaluation judge practically calls animal services.

Now that you've seen what's possible when you combine user behavior tracking with AI quality scoring, let's walk through how to add this same observability magic to your own multi-modal AI app.

## Your turn: See the complete picture

Want to add this observability magic to your own app? Here's how:

### 1. Install the packages

```bash lines wrap theme={null}
npm install @launchdarkly/observability
npm install @launchdarkly/session-replay
```

### 2. Initialize with observability

```javascript lines wrap theme={null}
import { initialize } from 'launchdarkly-js-client-sdk';
import Observability from '@launchdarkly/observability';
import SessionReplay from '@launchdarkly/session-replay';

const ldClient = initialize(clientId, user, {
  plugins: [
    new Observability(),
    new SessionReplay({
      privacySetting: 'strict' // Masks all data on the page - see https://launchdarkly.com/docs/sdk/features/session-replay-config#expand-javascript-code-sample
    })
  ]
});
```

### 3. Configure online evaluations in dashboard

<Frame caption="Install evaluation judges in your config">
  <img src="https://mintcdn.com/launchdarkly/hRg11GGWIRyNRMYL/images/tutorials/observability-multimodal-agents/install_judges.png?fit=max&auto=format&n=hRg11GGWIRyNRMYL&q=85&s=2c76782d0b08a4b46f870d06a79ac437" alt="Install Judges" width="2802" height="1650" data-path="images/tutorials/observability-multimodal-agents/install_judges.png" />
</Frame>

1. **Create your config in LaunchDarkly for LLM evaluation**
2. **Enable automatic accuracy scoring for production monitoring**

<Frame caption="Configure judges for accuracy scoring">
  <img src="https://mintcdn.com/launchdarkly/hRg11GGWIRyNRMYL/images/tutorials/observability-multimodal-agents/configure_judges.png?fit=max&auto=format&n=hRg11GGWIRyNRMYL&q=85&s=84bb9b61c75f1b44657803c93a1cc261" alt="Configure Judges" width="2860" height="776" data-path="images/tutorials/observability-multimodal-agents/configure_judges.png" />
</Frame>

3. **Set accuracy weight to 100% for production AI monitoring**
4. **Monitor your AI outputs with real-time evaluation scoring**

### 4. Connect the dots

**Session replay shows you:**

* Where users drop off
* What confuses them
* When they rage click
* How long they wait

**Online evaluations show you:**

* AI decision accuracy scores
* Why certain outputs scored low
* Pattern of good vs bad castings
* Safety concerns (even for pixels!)

Together they reveal the complete story of your AI app.

### Resources to get started:

**[Full Implementation Guide](https://github.com/launchdarkly-labs/scarlett-critter-casting)** - See how this pet app implements both features

**[Session Replay Tutorial](/docs/tutorials/detecting-user-frustration-session-replay)** - Official LaunchDarkly guide for detecting user frustration

**[When to Add Online Evals](/docs/tutorials/when-to-add-online-evals)** - Learn when and how to implement AI evaluation

The real magic is in having observability AND online evaluations.

## Try it yourself

**Cast your pet:** [https://scarlett-critter-casting.onrender.com/](https://scarlett-critter-casting.onrender.com/)

See your evaluation score ⭐. Understand why your cat is a shepherd and your dog is an angel. The AI has spoken, and now you can see exactly how much to trust it!

***

## Ready to add AI observability to your multi-modal agents?

Don't let your AI operate in the dark this holiday season. Get complete visibility into your multi-modal AI systems with LaunchDarkly's online evaluations and session replay.

<Tip>
  **Get started:** [Sign up for a free trial](https://app.launchdarkly.com/signup) → [Create your first config](/docs/home/agentcontrol/create) → Enable session replay and online evaluations → Ship with confidence.
</Tip>

## Further reading

**LaunchDarkly resources:**

* [AgentControl Quickstart Guide](/docs/home/agentcontrol/quickstart)
* [Online Evaluations](/docs/home/agentcontrol/online-evaluations)
* [Session Replay Documentation](/docs/home/observability/session-replay)

**Related tutorials:**

* [Detecting User Frustration with Session Replay](/docs/tutorials/detecting-user-frustration-session-replay)
* [Building Multi-Agent Systems with LangGraph](/docs/tutorials/agents-langgraph)
* [When to Add Online Evaluations](/docs/tutorials/when-to-add-online-evals)
* [Beyond n8n for Workflow Automation: Agent Graphs](/docs/tutorials/agent-graphs) - Per-node latency and quality monitoring on the graph itself
* [Offline Evaluation of RAG-Grounded Answers](/docs/tutorials/offline-evals) - Pair production observability with offline regression testing
* [Evaluate LLM code generation with LLM-as-judge evaluators](/docs/tutorials/custom-evals-claude-code) - Custom judges for code-generation workflows