psychometric_modeling

CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation

Authors: Mengting Chen, Yanshu Sun, Wanting Liang, Beidi Luan, Rui Sun, Dezhi Chen, Jing Li, Zuo Bai
Venue / Source: arXiv (2026)

Evaluating AI models using long-form answers is difficult because expert review is expensive and existing automated rules are often too rigid.

Executive Summary (TL;DR)

In plain words In the paper's terms

Evaluating AI models using long-form answers is difficult because expert review is expensive and existing automated rules are often too rigid. This paper introduces a new system that automatically selects the most informative evaluation criteria based on the specific type of task being performed and the consistency of the evaluators. This approach allows for accurate model ranking using far fewer criteria than previous methods, making the evaluation process both more efficient and reliable.

This paper presents CalibratedRubric, a framework that replaces deterministic rubric filtering with probabilistic, task-adaptive measurement to improve open-ended LLM evaluation. The method utilizes Bayesian calibration for judge reliability and item response theory (IRT) to select an optimal, compact rubric bank. Empirical results across four benchmarks demonstrate that the system maintains high rank fidelity while significantly reducing the number of rubrics required for robust scoring.

Authors' abstract (original)

Reliable evaluation of open-ended LLM outputs requires fine-grained rubrics, yet expert curation is costly and difficult to scale. Existing automated pipelines rely on strict judge unanimity and binary variance filters, which cannot distinguish measurable rubrics from informative ones. We introduce CalibratedRubric, a task-adaptive framework that combines type-specific scoring, Bayesian rubric-measurability filtering, and item response theory (IRT)-based bank assembly. CalibratedRubric estimates each rubric's measurability with a Beta--Bernoulli agreement posterior and uses a submodular information-coverage objective to construct compact rubric banks over the observed capability range. Across financial, healthcare, general, and legal benchmarks, measurability filtering improves human-gold agreement on JudgmentBench from $κ=0.604$ to $0.743$. IRT-based greedy selection improves cross-fitted rank fidelity over random selection across all six evaluated response blocks and requires only 49 rather than 131 rubrics to reach the target correlation on FinResearchBench decision-support tasks. Task-label perturbations further reduce system separation, confirming the practical relevance of task-adaptive scoring. These results support CalibratedRubric as an efficient, uncertainty-aware approach to open-ended LLM evaluation, with calibration gains depending on sufficient judge redundancy.

Quality & Relevance Evaluation Breakdown

Novelty

7.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

8.0/10

Consensus Challenge

5.0/10

Topic Relevance

8.0/10

Relevance for Educators & Research

Provides a methodology for handling item parameter drift and rater drift in AI-augmented assessment.
Offers a framework for 'AI-resistant' or 'AI-resilient' assessment by using psychometric models to identify which rubric items remain informative.
Contributes to the discussion on valid assessment design under generative AI by formalizing the 'rubric bank' construction process.

Methodology and Dataset

The study evaluates the framework across 534 tasks and 7,959 rubrics spanning four benchmarks: FinResearchBench-v2 (15 systems, 3 LLM judges), HealthBench, HelloBench (each with 6 systems, 4 LLM judges), and JudgmentBench (1,314 outputs, 5-member human/LLM panel). Models tested include various proprietary and open LLMs. Baselines include uniform binary scorers and random rubric selection.

Why Worth Reading

It provides a rigorous statistical foundation for improving the scalability and validity of LLM evaluation, moving away from flawed heuristic filters toward psychometrically grounded methods like IRT that are essential for handling the noise of automated or semi-automated assessment.

Key Claims & Verified Quotes

No verified quotes available for this paper.

Limitations & Study Design Flags

Monolingual: not mentioned in the text

The 2PL model identifiability is weak on small leaderboards, requiring regularization.
The assumption of local independence in IRT may not hold given shared dependencies in LLM outputs.
Systematic judge-human mismatches persist even after measurability filtering.
Rubric-level calibration performs poorly in two-judge scenarios due to insufficient redundancy.

Model inference (gaps & contradictions)

[method / theoretical]: The authors acknowledge that while they use the 2PL model for rubric selection, the evidence for universal 2PL identifiability in these LLM evaluation contexts is limited.

#IRT #automated_assessment #psychometrics #LLM_as_a_Judge #rubric_validity