Online Evaluations is now Generally Available for AgentControl, enabling automated, model-based evaluation of LLM completions in both production and pre-production environments. This GA release also introduces Customizable Judges, allowing you to define evaluation prompts and scoring criteria aligned with your domain-specific requirements.

Online Evaluations attaches AI Judges to config variations and emits structured evaluation metrics (for example, accuracy, relevance, or toxicity) for each completion. Judge results are surfaced in real time on the AI Config Monitoring page and can be used to detect regressions when prompts, models, or parameters change.

Customizable Judges give you fine-grained control over evaluation logic while maintaining consistent metric output for monitoring and analysis.

Docs