AI in Education
UK EN

Papers tagged with: #llm

8.7
psychometric_modeling

LLM Judges as Raters: A Pre-Registered Audit of Severity, Halo, Reliability, and Version Instability in LLM Essay Scoring on Public Corpora

Veerendra Kumar Sunkavalli · arXiv (2026) · doi:10.48550/arxiv.2608.29517 · gemini-3.1-flash-lite

Read full analysis →

8.0
assessment_design

A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation

Jianlin Chen, Wenhui Chen, Ziyao Lin, Chi Man Vong · arXiv (2026) · doi:10.48550/arxiv.2608.24419 · gemini-3.1-flash-lite

Read full analysis →

7.5
assessment_design

Synthetic Worldview Reconstruction (SWR): A Methodological Framework for LLM-Assisted Group-Level Belief System Analysis

Lukas Geiger · Zenodo (CERN European Organization for Nuclear Research) (2026) · doi:10.5281/zenodo.18736720 · gemini-3.1-flash-lite

Read full analysis →

7.5
psychometric_modeling

A Dual-Dimensional LLM Framework for Automated Item Incidental Content Similarity Analysis in Large-Scale Assessments

Jing Huang, Jihong Zhang, Hua-Hua Chang · arXiv (2026) · doi:10.48550/arxiv.2608.24825 · gemini-3.1-flash-lite

Read full analysis →

7.5
psychometric_modeling

Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation

Nishant Balepur, Paiheng Xu, Wei Ai, Eunsol Choi, Rachel Rudinger, Jordan Boyd-Graber · arXiv (2026) · doi:10.48550/arxiv.2608.29887 · gemini-3.1-flash-lite

Read full analysis →

7.7
psychometric_modeling

Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory

Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim · arXiv (2026) · doi:10.48550/arxiv.2602.00521 · gemini-3.7-flash

Read full analysis →

7.5
psychometric_modeling

Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases

Davood Wadi, Mohsen Ghodrat, Matthew Philp · arXiv (2026) · doi:10.48550/arxiv.2608.10503 · gemini-3.1-flash-lite

Read full analysis →

7.5
psychometric_modeling

Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks

Xiao Fei, Yang Zhang, Sarah Almeida Carneiro, Michalis Vazirgiannis · arXiv (2026) · doi:10.48550/arxiv.2608.02966 · gemini-3.1-flash-lite

Read full analysis →

7.5
psychometric_modeling

An Interpretable and Scalable Framework for Evaluating Large Language Models

Xinhao Qu, Qiang Heng, Hao Zeng, Xiaoqian Liu · arXiv (2026) · doi:10.48550/arxiv.2605.07046 · gemini-3.7-flash

Read full analysis →

7.5
psychometric_modeling

Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty

Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Fabian Zehner, Hendrik Drachsler, Ulf Kroehne · arXiv (2026) · doi:10.48550/arxiv.2610.02877 · gemini-3.1-flash-lite

Read full analysis →

7.4
psychometric_modeling

Interpretable, Fairly Evaluated Automated L2 Speaking Assessment that Beats the Single-Human Ceiling and Why Pause Encoding Does Not Change LLM Fluency Scores

Eichi Uehara · arXiv (2026) · doi:10.48550/arxiv.2608.26137 · gemini-3.8-flash

Read full analysis →

7.4
psychometric_modeling

Machine--human agreement in marking undergraduate assignments is specific to model, prompt and institution

Giulio Corsi, Maryam Abo-Tabik, Yael Benn, Georgiana Thorpe Apreutesei, Roni Tibon, Alexandru Marcoci, Deborah Talmi · arXiv (2026) · doi:10.48550/arxiv.2610.18019 · gemini-3.8-flash

Read full analysis →

7.8
assessment_design

Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges

Saeid Asgari, Rui Ying, Sara Malvar, Rakshanda Agarwal, Bruce Sun, Claire Wang, Leonardo de Oliveira Nunes, Ranveer Chandra · arXiv (2026) · doi:10.48550/arxiv.2610.24971 · gemini-3.8-flash

Read full analysis →

7.4
psychometric_modeling

Scalable Text-Embedding-informed Cognitive Diagnosis of Large Language Models

Jia Liu, Zhiyu Xu, Yuqi Gu · arXiv (2026) · doi:10.48550/arxiv.2603.14676 · gemini-3.7-flash

Read full analysis →

7.2
assessment_design

MS-Exam-Gen: Source-Grounded Benchmark Construction for Evaluating LLMs on Textual Multiple Sclerosis MRI Knowledge

Abdul Basit, Muhammad Abdullah Hanif, Muhammad Shafique · arXiv (2026) · doi:10.48550/arxiv.2610.06170 · gemini-3.1-flash-lite

Read full analysis →

8.0
psychometric_modeling

Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

Delip Rao, Chris Callison-Burch · arXiv (2026) · doi:10.48550/arxiv.2606.00093 · gemini-3.8-flash

Read full analysis →

7.3
psychometric_modeling

Auditing LLM Benchmarks with Item Response Theory

Sander Land, Daniel M. Bikel · arXiv (2026) · doi:10.48550/arxiv.2605.30504 · gemini-3.6-flash

Read full analysis →

7.3
psychometric_modeling

From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment

Ivo Bueno, Babette Bühler, Philipp Stark, Tim Fütterer, Ulrich Trautwein, Dorottya Demszky, Heather Hill, Enkelejda Kasneci · arXiv (2026) · doi:10.48550/arxiv.2606.05180 · gemini-3.6-flash

Read full analysis →

7.3
psychometric_modeling

Algorithmic drift in generative adaptive learning

Firuz Kamalov, Hana Sulieman, Ahmed El Sayed, Sivaraj R., Nitesh Sughnani · arXiv (2026) · doi:10.48550/arxiv.2610.31041 · gemini-3.7-flash

Read full analysis →

7.3
psychometric_modeling

Which AI model is truly the best: how reliable are public benchmarks of frontier AI models?

Miloš Kankaraš · Zenodo (CERN European Organization for Nuclear Research) (2026) · doi:10.5281/zenodo.22998204 · gemini-3.7-flash

Read full analysis →

7.2
psychometric_modeling

Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation

Gili Lior, Tzviel Frostig, Gabriel Stanovsky, Matan Eyal · arXiv (2026) · doi:10.48550/arxiv.2606.15643 · gemini-3.7-flash

Read full analysis →

7.2
psychometric_modeling

CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation

Mengting Chen, Yanshu Sun, Wanting Liang, Beidi Luan, Rui Sun, Dezhi Chen, Jing Li, Zuo Bai · arXiv (2026) · doi:10.48550/arxiv.2607.29252 · gemini-3.1-flash-lite

Read full analysis →

7.4
assessment_design

MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring

Woojin Lee, Yoonkyu Woo, Jinah Kim, Hyeok-Min Gwon, Jin‐Xia Huang · arXiv (Cornell University) (2026) · doi:10.48550/arxiv.2610.31679 · gemini-3.6-flash

Read full analysis →

7.2
psychometric_modeling

Operating characteristics of agreement metrics in AI-based scoring: a Monte Carlo simulation

Alperen Yandi · Frontiers in Psychology (2026) · doi:10.3389/fpsyg.2026.1705653 · gemini-3.8-flash

Read full analysis →

7.9
quasi_experiment

Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items

Peng Xie, Amr Alanwar · arXiv (2026) · doi:10.48550/arxiv.2610.40241 · gemini-3.8-flash

Read full analysis →

7.2
assessment_design

Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revise

Keno Harada, Lui Yoshida, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo · Conference on Computational Natural Language Learning (2026) · doi:10.18653/v1/2026.conll-main.47 · gemini-3.6-flash

Read full analysis →