assessment_design
A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation
Jianlin Chen, Wenhui Chen, Ziyao Lin, Chi Man Vong · arXiv (2026) · doi:10.48550/arxiv.2608.24419 · gemini-3.1-flash-lite
Papers tagged with: #llm-as-a-judge
psychometric_modeling
Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim · arXiv (2026) · doi:10.48550/arxiv.2602.00521 · gemini-3.7-flash
psychometric_modeling
Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty
Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Fabian Zehner, Hendrik Drachsler, Ulf Kroehne · arXiv (2026) · doi:10.48550/arxiv.2610.02877 · gemini-3.1-flash-lite
assessment_design
Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges
Saeid Asgari, Rui Ying, Sara Malvar, Rakshanda Agarwal, Bruce Sun, Claire Wang, Leonardo de Oliveira Nunes, Ranveer Chandra · arXiv (2026) · doi:10.48550/arxiv.2610.24971 · gemini-3.8-flash
psychometric_modeling
Auditing LLM Benchmarks with Item Response Theory
Sander Land, Daniel M. Bikel · arXiv (2026) · doi:10.48550/arxiv.2605.30504 · gemini-3.6-flash
psychometric_modeling
CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
Mengting Chen, Yanshu Sun, Wanting Liang, Beidi Luan, Rui Sun, Dezhi Chen, Jing Li, Zuo Bai · arXiv (2026) · doi:10.48550/arxiv.2607.29252 · gemini-3.1-flash-lite