Online Evaluations for AI Configs (Beta)

Online Evaluations are now available in Production, enabling automatic quality assessment of AI responses directly within LaunchDarkly. Teams can attach built-in judges — Accuracy, Relevance, and Toxicity — to AI Config variations with configurable sampling rates. Evaluation metrics are emitted as custom events and automatically added as metrics for use in A/B tests and Guarded Releases.

This allows teams to continuously monitor AI response quality in real time, track key metrics, and identify regressions or degradation based on live production data.

In Beta — Judges must be installed before being attached to AI Config variations. Currently, judges are non-configurable aside from provider and model selection.

Learn more

• AI SDK
• OpenAI Provider
• LangChain Provider
• Vercel AI Provider