assessment_design

Proxy Confidence: Auditing Black-Box LLM Agents with a Surrogate's Log-Probabilities

Authors: Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra
Venue / Source: arXiv (2026)

A small, open-weight sidecar model can audit a black-box LLM agent's tool calls and code generation with better reliability than the agent's own confidence.

Key takeaways

A small, open-weight model running in parallel can successfully audit the tool-call reliability of frontier-scale black-box agents.
The method uses generative likelihoods for argument-value accuracy and discriminative verdicts for holistic correctness, creating a robust, regime-dependent safety system.
Proxy confidence significantly outperforms the agent's own self-reported confidence, particularly in regimes where the agent is highly overconfident.
The sidecar strategy is computationally inexpensive, relying on a single forward pass over an existing action, and does not require fine-tuning or internal agent access.
The efficacy of feedback signals depends on whether the environment provides sufficient diagnostic error information, highlighting the need for rigorous control baselines like random-value arms.

Executive Summary (TL;DR)

In plain words In the paper's terms

AI agents often make silent errors when using tools, and their own confidence scores are frequently unreliable. This study shows that running a smaller, open-source AI model alongside the main agent to verify its outputs—by analyzing how likely the small model thinks the actions are—is a cheap and effective way to catch these errors. This approach helps developers build more trustworthy AI systems without needing access to the main agent's internal settings.

This paper introduces a training-free sidecar mechanism to audit black-box LLM agents using log-probabilities from smaller, open-weight surrogate models. It demonstrates that combining discriminative verdict readouts with generative likelihoods reliably ranks action correctness across diverse domains. The method outperforms both the agent's internal self-assessment and self-consistency baselines, especially on confident mistakes and in near-deterministic deployment regimes.

Authors' abstract (original)

A deployed LLM agent emits tool calls, queries, and code that can be silently wrong -- by the time the error surfaces, the action has run. Frontier chat APIs hide the model's token probabilities; the agent's stated confidence barely beats chance on the mistakes that matter; and resampling does not help, since frontier models are highly repetitive, reproducing the same call across samples. We recover the missing signal from a low-cost open-weight surrogate run in parallel. It reads the same context, schema, and proposed action as the agent, then scores the call from its own log-probabilities through a family of complementary readouts: teacher forcing and request-PMI weigh the likelihood of each argument value, a discriminative verdict judges the call as a whole, and tool-choice competition tests the function against its siblings. One principle says which to trust: a generative likelihood localizes wrong argument values, while the verdict catches holistically wrong calls. When the error type is unknown, an ensemble is the low-regret default. The readout is training-free, needs no access to the agent's internals, and costs one prefill pass alongside the tool call. On difficult coding tasks it reaches AUROC 0.825 where the actor's stated confidence is near chance (0.598), and the generative readouts beat it by +0.07 to +0.28 across three further actors. Against self-consistency it gains +0.14 to +0.19 on near-deterministic actors, at 1/K the cost. The signal drives two deployment modes: a real-time gate escalating the least-trustworthy calls for review (+0.05 to +0.30 accepted-action accuracy at 50% coverage), and confidence feedback, returning the tool result with the score so the agent adapts its next step -- lifting task success on live-execution benchmarks (+0.119 and +0.137, p <= 1e-4) and beating a random-value control where step errors are silent (+0.078, p = 0.003).

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

8.0/10

Consensus Challenge

5.0/10

Topic Relevance

8.0/10

Relevance for Educators & Research

Provides a quantifiable method for auditing agent performance in high-stakes environments.
Offers a framework for 'AI-resistant' assessment design by gating tool-use for accuracy.
Demonstrates how to handle item parameter drift and model reliability in black-box scenarios.

Methodology and Dataset

The study evaluates the proxy-confidence sidecar across five actor families (including frontier closed and open models) using a dataset of tool-calling and code-generation benchmarks (e.g., AppWorld, BFCL, LiveCodeBench, BIRD). The auditors are Qwen3-32B, Qwen3-14B, and Gemma-3-27B. Evaluated datasets contain thousands of calls, with labels derived from objective execution/AST/gold-match checks rather than LLM judgments.

Why Worth Reading

This paper provides a scalable, engineering-focused solution to the 'silent failure' problem in AI agent tool-use, replacing reliance on opaque self-assessment with verifiable proxy confidence. It offers an actionable framework for educators to evaluate AI-agent integrity in computational assessments by providing a method to detect and gate incorrect tool outputs.

Key Claims & Verified Quotes

The proxy-confidence sidecar effectively predicts action correctness for black-box LLM agents.

Quote: “This proxy ranks correct calls above wrong ones more reliably than either signal a black-box operator has today.” — 10.48550/arxiv.2610.03894, p. 1, para. 12 (exact)

Limitations & Study Design Flags

The method requires actions that are decidable by 'reading' them; it does not solve errors involving missing external facts.
The orientation sign (correlation between signal and correctness) must be fit on held-out labels per regime.
The 'beats-self' edge is regime-graded, not universal; on some benchmarks, the proxy signal only ties the agent’s self-assessment.
Multi-tenant serving and scorer sizes beyond the 2x ratio tested are not profiled.

From the article text (quote verified)

supports Self-consistency (sampling) collapses for near-deterministic agents. «The actor re-emits the identical call on the overwhelming majority of inputs, leaving almost no sample variance to read», p. 2, para 1

Model inference (gaps & contradictions)

[Method / Data dependency]: While the proxy method improves accuracy, it still relies on a 'gold' check or execution environment to validate success, which may not always be available in student-facing generative AI applications.

#AI agents #Tool-use evaluation #Confidence estimation #Reliability #Black-box auditing