assessment_design
A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation
Jianlin Chen, Wenhui Chen, Ziyao Lin, Chi Man Vong · arXiv (2026) · doi:10.48550/arxiv.2608.24419 · gemini-3.1-flash-lite
Papers tagged with: #construct-validity
assessment_design
GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
Umesh Bodhwani, Thanh Tran, Kai Wei · arXiv (2026) · doi:10.48550/arxiv.2609.12191 · gemini-3.1-flash-lite
theoretical
Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs
Manuel Pita · arXiv (2026) · doi:10.48550/arxiv.2606.28574 · gemini-3.1-flash-lite
psychometric_modeling
Personality Without Persons? A Psychometric Critique of Big Five Testing in Large Language Models
Kim Zierahn, Cristina Cachero, Anna Korhonen, Nuria Oliver · arXiv (2026) · doi:10.48550/arxiv.2607.02325 · gemini-3.1-flash-lite
psychometric_modeling
Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim · arXiv (2026) · doi:10.48550/arxiv.2602.00521 · gemini-3.7-flash
psychometric_modeling
Interpretable, Fairly Evaluated Automated L2 Speaking Assessment that Beats the Single-Human Ceiling and Why Pause Encoding Does Not Change LLM Fluency Scores
Eichi Uehara · arXiv (2026) · doi:10.48550/arxiv.2608.26137 · gemini-3.8-flash
psychometric_modeling
Machine--human agreement in marking undergraduate assignments is specific to model, prompt and institution
Giulio Corsi, Maryam Abo-Tabik, Yael Benn, Georgiana Thorpe Apreutesei, Roni Tibon, Alexandru Marcoci, Deborah Talmi · arXiv (2026) · doi:10.48550/arxiv.2610.18019 · gemini-3.8-flash
assessment_design
Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges
Saeid Asgari, Rui Ying, Sara Malvar, Rakshanda Agarwal, Bruce Sun, Claire Wang, Leonardo de Oliveira Nunes, Ranveer Chandra · arXiv (2026) · doi:10.48550/arxiv.2610.24971 · gemini-3.8-flash
psychometric_modeling
Inside the Scorer: Mechanistic Evidence on Rubric Control in LLM-Based Automated Essay Scoring from 25 Base Models
Chungche Huang, Po-Ya Angela Wang, Shu-Kai Hsieh · arXiv (2026) · doi:10.48550/arxiv.2610.25724 · gemini-3.7-flash