GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
Authors: Umesh Bodhwani, Thanh Tran, Kai Wei
Venue / Source: arXiv (2026)
Subjective user satisfaction ratings are decorrelated from objective task success, meaning that AI-driven release gates often promote agents that satisfy users while failing their tasks.
Key takeaways
- User satisfaction ratings show no meaningful correlation with actual task success, indicating that subjective feedback is an unreliable metric for evaluating AI agent performance.
- While LLM-as-a-judge approaches demonstrate robust ranking validity for agents with distinct capability gaps, they exhibit high disagreement rates when evaluating agents of similar performance levels.
- A judge-free completion indicator serves as a reliable, cost-effective mechanism for detecting significant functional regressions in automated assessment workflows.
- The study is limited by its reliance on synthetic environments, meaning the findings may not fully generalize to the complexities of live, unconstrained user interactions.
Executive Summary (TL;DR)
Authors' abstract (original)
Comparing and selecting task-oriented LLM agents increasingly relies on a low-cost offline evaluation gate: persona-driven LLM user-simulators converse with each candidate, an LLM-as-a-judge scores the transcripts, and the higher-scoring agent is promoted. We introduce GAUGE, a reusable offline protocol that measures whether this gate's ranking matches a grounded verifiable reward across 25 agents from six providers on the $τ^2$-bench and SimulatorArena benchmarks, separating two kinds of evaluation validity that release practices conflate: ranking validity and construct validity. First, a satisfaction-success gap: satisfaction carries essentially no information about task success, as conversations rated satisfied by our blind panel are decorrelated from actual success, with 57.5% of them failing the customer's task, a pattern consistent across five rater populations, both benchmarks, and every subjective dimension we rated. Second, while the gate's ranking is robust across the broad capability span, it loses resolution among the near-equal strong agents: this decision-disagreement rate jumps from $<$1% on wide-reward pairs to 31% on close pairs. The gate is thus human-validated yet mis-anchored. As a remedy, we propose a calibrate-then-trust cadence in which a judge-free completion bit is a zero-cost tripwire for truncation regressions.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Provides a framework for evaluating the validity of LLM-based assessment tools.
- Demonstrates how to quantitatively detect 'construct-gap' using controlled perturbations.
- Offers a strategy (calibrate-then-trust) for balancing cost and validity in high-stakes automated evaluation.
- Explains how to use non-LLM verifiable rewards to check for grade inflation or metric gaming.
Methodology and Dataset
The study evaluates 25 agents across six providers (Anthropic, Meta, Mistral, Qwen, DeepSeek, OpenAI) using 3,700 transcripts. Primary datasets include τ2-bench (retail/airline) and SimulatorArena (math tutoring). It employs a 3-person blind human panel for grounding, and four distinct LLM judges for comparison.
Why Worth Reading
It provides a foundational, rigorous critique of the 'LLM-as-a-judge' paradigm, offering a quantifiable psychometric distinction between ranking validity and construct validity that is essential for educators designing or evaluating AI-driven assessment tools.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Small sample: not found
- Unrepresentative sample: not mentioned in the text
- Monolingual: not mentioned in the text
- The human panel is limited to 150 transcripts, though results are robust to leave-one-annotator-out analysis.
- Controlled-degradation set relies on budget-style perturbations (truncation) which may not capture all real-world semantic failure modes.
- Generalization to live, non-synthetic user interactions remains empirically untested.
- Results from τ2-bench may not be leaderboard-comparable due to the harness's specific parser and Bedrock runtime.
- [metric / Construct validity failure]: The paper highlights that educators and developers conflate ranking validity (ordering systems) with construct validity (measuring the correct outcome), a classic psychometric trap.
- [method / Metric optimization]: It validates Goodhart's Law: optimizing for a satisfaction proxy systematically causes the metric to diverge from the actual objective (task completion).