Papers tagged with: #validation

8.0
assessment_design

Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks

Michael Kouremetis, Ads Dawson, Raja Sekhar Rao Dheekonda, Brian Greunke · arXiv (2026) · doi:10.48550/arxiv.2607.21763 · gemini-3.1-flash-lite

Read full analysis →

8.7
psychometric_modeling

LLM Judges as Raters: A Pre-Registered Audit of Severity, Halo, Reliability, and Version Instability in LLM Essay Scoring on Public Corpora

Veerendra Kumar Sunkavalli · arXiv (2026) · doi:10.48550/arxiv.2608.29517 · gemini-3.1-flash-lite

Read full analysis →

8.0
assessment_design

A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation

Jianlin Chen, Wenhui Chen, Ziyao Lin, Chi Man Vong · arXiv (2026) · doi:10.48550/arxiv.2608.24419 · gemini-3.1-flash-lite

Read full analysis →

7.7
assessment_design

GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents

Umesh Bodhwani, Thanh Tran, Kai Wei · arXiv (2026) · doi:10.48550/arxiv.2609.12191 · gemini-3.1-flash-lite

Read full analysis →

8.0
detector_evaluation

StyleShield: Exposing the Fragility of AIGC Detectors through Continuous Controllable Style Transfer

Guantian Zheng · arXiv (2026) · doi:10.48550/arxiv.2605.00924 · gemini-3.1-flash-lite

Read full analysis →

8.0
psychometric_modeling

Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks

Juan Francisco, Mandujano Reyes · arXiv (2026) · doi:10.48550/arxiv.2607.25257 · gemini-3.1-flash-lite

Read full analysis →

7.7
theoretical

Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs

Manuel Pita · arXiv (2026) · doi:10.48550/arxiv.2606.28574 · gemini-3.1-flash-lite

Read full analysis →

7.6
psychometric_modeling

Personality Without Persons? A Psychometric Critique of Big Five Testing in Large Language Models

Kim Zierahn, Cristina Cachero, Anna Korhonen, Nuria Oliver · arXiv (2026) · doi:10.48550/arxiv.2607.02325 · gemini-3.1-flash-lite

Read full analysis →

7.7
psychometric_modeling

Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory

Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim · arXiv (2026) · doi:10.48550/arxiv.2602.00521 · gemini-3.7-flash

Read full analysis →

7.5
psychometric_modeling

Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks

Xiao Fei, Yang Zhang, Sarah Almeida Carneiro, Michalis Vazirgiannis · arXiv (2026) · doi:10.48550/arxiv.2608.02966 · gemini-3.1-flash-lite

Read full analysis →

7.5
assessment_design

Do Coding Agents Deceive Us? Detecting and Preventing Cheating via Capped Evaluation with Randomized Tests

Thanawat Lodkaew, Johannes Ackermann, Soichiro Nishimori, Nontawat Charoenphakdee, Masashi Sugiyama, Takashi Ishida · arXiv (2026) · doi:10.48550/arxiv.2606.07379 · gemini-3.1-flash-lite

Read full analysis →

7.5
assessment_design

NeutronGym: Physics-Graded Neutron Instrument Design for LLM Agents

Lijie Ding, Changwoo Do · arXiv (2026) · doi:10.48550/arxiv.2610.03631 · gemini-3.1-flash-lite

Read full analysis →

7.5
psychometric_modeling

Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty

Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Fabian Zehner, Hendrik Drachsler, Ulf Kroehne · arXiv (2026) · doi:10.48550/arxiv.2610.02877 · gemini-3.1-flash-lite

Read full analysis →

7.5
psychometric_modeling

Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?

Qiaoyuan Zheng, Yiqu Yang · arXiv (2026) · doi:10.48550/arxiv.2609.00482 · gemini-3.1-flash-lite

Read full analysis →

7.4
psychometric_modeling

Interpretable, Fairly Evaluated Automated L2 Speaking Assessment that Beats the Single-Human Ceiling and Why Pause Encoding Does Not Change LLM Fluency Scores

Eichi Uehara · arXiv (2026) · doi:10.48550/arxiv.2608.26137 · gemini-3.8-flash

Read full analysis →

7.5
psychometric_modeling

From Feature-Based Models to Generative AI: Validity Evidence for Constructed Response Scoring

Jodi M. Casabianca, Daniel F. McCaffrey, Matthew S. Johnson, Naim Alper, Vladimir Zubenko · arXiv (2026) · doi:10.48550/arxiv.2603.19280 · gemini-3.8-flash

Read full analysis →

7.4
psychometric_modeling

Machine--human agreement in marking undergraduate assignments is specific to model, prompt and institution

Giulio Corsi, Maryam Abo-Tabik, Yael Benn, Georgiana Thorpe Apreutesei, Roni Tibon, Alexandru Marcoci, Deborah Talmi · arXiv (2026) · doi:10.48550/arxiv.2610.18019 · gemini-3.8-flash

Read full analysis →

7.8
assessment_design

Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges

Saeid Asgari, Rui Ying, Sara Malvar, Rakshanda Agarwal, Bruce Sun, Claire Wang, Leonardo de Oliveira Nunes, Ranveer Chandra · arXiv (2026) · doi:10.48550/arxiv.2610.24971 · gemini-3.8-flash

Read full analysis →

7.4
psychometric_modeling

What exam scores can and cannot prove about unauthorized AI assistance: Evidence from a highly public classroom episode

Chad M. Topaz, Utsav Bahl · arXiv (2026) · doi:10.48550/arxiv.2607.27978 · gemini-3.1-flash-lite

Read full analysis →

7.3
psychometric_modeling

Representational Control over Self-Report & Behavior Coherence in LLM Risk-Taking

Rafal Kocielnik, Peiyang Song, Pengrui Han, Myrl G. Marmarelis, Ramit Debnath, Dean Mobbs, R. Michael Alvarez · arXiv (2026) · doi:10.48550/arxiv.2610.04125 · gemini-3.1-flash-lite

Read full analysis →

7.3
psychometric_modeling

Identifying Causes of Test Unfairness: Manipulability and Separability

Youmi Suk, Weicong Lyu · arXiv (2026) · doi:10.48550/arxiv.2601.13449 · gemini-3.1-flash-lite

Read full analysis →

8.0
psychometric_modeling

Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

Delip Rao, Chris Callison-Burch · arXiv (2026) · doi:10.48550/arxiv.2606.00093 · gemini-3.8-flash

Read full analysis →

7.4
psychometric_modeling

Inside the Scorer: Mechanistic Evidence on Rubric Control in LLM-Based Automated Essay Scoring from 25 Base Models

Chungche Huang, Po-Ya Angela Wang, Shu-Kai Hsieh · arXiv (2026) · doi:10.48550/arxiv.2610.25724 · gemini-3.7-flash

Read full analysis →

7.3
psychometric_modeling

Auditing LLM Benchmarks with Item Response Theory

Sander Land, Daniel M. Bikel · arXiv (2026) · doi:10.48550/arxiv.2605.30504 · gemini-3.6-flash

Read full analysis →

7.3
assessment_design

Prompt Sensitivity in LLM-Based Essay Scoring is Model-Specific

Giulio Corsi, Alexandru Marcoci, Maryam Abo-Tabik, Georgiana Thorpe Apreutesei, Lyba Razzaq, Roni Tibon, Yael Benn, Deborah Talmi · CLOK (University of Central Lancashire) (2026) · doi:10.5281/zenodo.21039771 · gemini-3.7-flash

Read full analysis →

7.2
detector_evaluation

Why AI Detection Fails for Academic Integrity

Jonathan A. Karr, Grigorii Khvatskii, Ting Hua, Nitesh V. Chawla · arXiv (2026) · doi:10.48550/arxiv.2608.11256 · gemini-3.1-flash-lite

Read full analysis →

7.2
psychometric_modeling

Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation

Gili Lior, Tzviel Frostig, Gabriel Stanovsky, Matan Eyal · arXiv (2026) · doi:10.48550/arxiv.2606.15643 · gemini-3.7-flash

Read full analysis →

7.2
psychometric_modeling

CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation

Mengting Chen, Yanshu Sun, Wanting Liang, Beidi Luan, Rui Sun, Dezhi Chen, Jing Li, Zuo Bai · arXiv (2026) · doi:10.48550/arxiv.2607.29252 · gemini-3.1-flash-lite

Read full analysis →

7.4
assessment_design

MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring

Woojin Lee, Yoonkyu Woo, Jinah Kim, Hyeok-Min Gwon, Jin‐Xia Huang · arXiv (Cornell University) (2026) · doi:10.48550/arxiv.2610.31679 · gemini-3.6-flash

Read full analysis →

7.2
psychometric_modeling

Operating characteristics of agreement metrics in AI-based scoring: a Monte Carlo simulation

Alperen Yandi · Frontiers in Psychology (2026) · doi:10.3389/fpsyg.2026.1705653 · gemini-3.8-flash

Read full analysis →

7.9
quasi_experiment

Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items

Peng Xie, Amr Alanwar · arXiv (2026) · doi:10.48550/arxiv.2610.40241 · gemini-3.8-flash

Read full analysis →

7.2
assessment_design

Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revise

Keno Harada, Lui Yoshida, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo · Conference on Computational Natural Language Learning (2026) · doi:10.18653/v1/2026.conll-main.47 · gemini-3.6-flash

Read full analysis →

7.2
psychometric_modeling

Reducing Differential Item Functioning via Process Data

Ling Chen, Susu Zhang, Jingchen Liu · arXiv (2025) · doi:10.48550/arxiv.2504.00136 · gemini-3.1-flash-lite

Read full analysis →