psychometric_modeling

Operating characteristics of agreement metrics in AI-based scoring: a Monte Carlo simulation

Authors: Alperen Yandi
Venue / Source: Frontiers in Psychology (2026)

Common reliability metrics like ICC and QWK can pass AI scoring models as adequate even when systematic subgroup bias exists.

Key takeaways

ICC(A,1) and quadratic weighted kappa routinely exceed acceptable reliability thresholds even when systematic group bias is present.
Krippendorff's ordinal alpha behaves overly conservatively under AI scoring distortions, rarely attaining standard adequacy thresholds.
Real-world evaluations of ChatGPT and Gemini on essay grading mirrored simulated defects, including substantial negative bias and midpoint compression.
Automated scoring frameworks must adopt multi-metric diagnostics, pairing agreement metrics with Bland-Altman difference plots and tolerance bands.

Executive Summary (TL;DR)

In plain words In the paper's terms

When using AI tools to grade open-ended student work, educators often rely on standard agreement statistics to verify the AI's accuracy against human teachers. This study shows that widely used agreement metrics can easily rate an AI grader as acceptable even when it systematically undergrades certain groups or avoids extreme scores. The author recommends combining multiple diagnostic tools, such as Bland-Altman difference plots, rather than relying on a single reliability score.

Conventional agreement metrics used to validate automated scoring models react inconsistently to characteristic generative AI errors such as midpoint compression, variance inflation, and subgroup bias. Across 216 Monte Carlo conditions and 1,080,000 replications, ICC(A,1) and quadratic weighted kappa frequently surpassed adequacy benchmarks despite injected systematic subgroup biases and exhibited severe null liberality. In contrast, Krippendorff's alpha was overly conservative across distorted conditions, demonstrating that single-metric evaluations fail to detect operational invalidity in AI raters.

Authors' abstract (original)

Introduction: This study analyzed the threshold-exceedance performance of human and AI scoring agreement measures for scoring of open-ended items. Monte Carlo simulation was performed to represent the different types of errors encountered in automatic scoring determined by using studies in the literature. Accordingly, an examination was conducted of the agreement between human and artificial intelligence ratings under differing error conditions. The objective of the research was to ascertain the statistical power levels achieved by disparate agreement metrics under varying conditions. The objective of this research was to ascertain which metric would prove to be the most effective method, and under which conditions. Methods: Data with conditions including systematic additive bias, variance inflation, midpoint compression, class imbalance, and subgroup-related offsets. Human scores served as a reference to assess agreement. Agreement levels were evaluated with ICC(A,1), Krippendorff's α (ranked), quadratic weighted kappa (QWK), and Bland-Altman along with tolerance-based agreement metrics. Threshold-exceedance performance was defined as the proportion with which each metric surpassed conventional adequacy standards. Analyses were also conducted on real data to validate the analyses conducted in the second part of the study. In this part, written texts were scored by six different students using three teachers and two large language models. Results: ICC(A,1) shown higher threshold-exceedance performance for low and moderate variance inflation. QWK was observed to reach a moderate level of robustness. Krippendorff's a showed consistent performance, especially in conditions where the distributions were unbalanced or variance inflated. Tolerance-based fit demonstrated numerical closeness between human and AI scores. The findings showed patterns consistent with simulated impairments. Discussion: All findings indicate that fit indices vary systematically across different structural mechanisms and sampling conditions. The results suggest that different conditions can affect the interpretability of automated scores. Accordingly, the need for multi-metric assessment frameworks when assessing human-AI score fit is highlighted.

Quality & Relevance Evaluation Breakdown

Novelty

6.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

6.5/10

Consensus Challenge

7.0/10

Topic Relevance

8.0/10

Relevance for Educators & Research

Provides a direct statistical foundation for evaluating LLMs as raters in educational assessment.
Demonstrates psychometric methods to detect validity degradation, rater drift, and subgroup scoring disparities caused by generative AI.
Highlights specific metric vulnerabilities when evaluating AI resilience in performance assessments.

Methodology and Dataset

A 3 x 3 x 3 x 2 x 2 x 2 fully crossed Monte Carlo simulation across 216 conditions and 5,000 replications per cell (1,080,000 simulated evaluations per metric). Factors manipulated were sample size (N = 100, 300, 1,000), additive bias (-0.5, 0, +0.5), AI error variance (low = 0.40, mid = 0.50, high = 0.95), midpoint compression (lambda_c = 0.40), class imbalance, and subgroup fairness offset (delta = 0.30). Real-world empirical validation utilized 6 B2-level student essays scored on a 0-70 rubric by 3 human teachers and 2 large language models (ChatGPT and Gemini) with 2,000 bootstrap iterations.

Why Worth Reading

This paper demonstrates quantitatively why high inter-rater reliability coefficients (such as ICC or QWK >= 0.70) between human instructors and LLM raters do not guarantee valid or fair automated grading. For psychometricians and assessment leaders implementing LLM scoring, it provides concrete operating characteristics of classical metrics and offers an actionable, multi-metric diagnostic protocol including Bland-Altman limits of agreement and tolerance indices.

Key Claims & Verified Quotes

ICC(A,1) and QWK can easily exceed conventional adequacy cutoffs in the absence of true scoring agreement or despite subgroup bias.

Quote: “ICC(A,1) and QWK showed high threshold exceedances in the null scenario. This result demonstrates that the conventional adequacy limits of these metrics can be easily exceeded even in the absence of a true discrepancy.” — 10.3389/fpsyg.2026.1705653, p. 16, para. 9 (exact)

High random error variance in AI scoring crushes the probability of exceeding adequacy thresholds across all agreement metrics.

Quote: “When the AI score distribution exceeded the variability of human scores, threshold-exceedance performance dropped to nearly zero for all metrics.” — 10.3389/fpsyg.2026.1705653, p. 16, para. 13 (exact)

Limitations & Study Design Flags

Small sample: found
“Since sample size of real data as very small, all statistics were estimated more reliably with 2000 bootstrap replication.” (p. 7)
Unrepresentative sample: found
“Six B2-level student essays were obtained from a previously completed instructional task.” (p. 6)
Monolingual: not mentioned in the text
Single detector: not mentioned in the text
No adversarial testing: not mentioned in the text

The simulation assumed constant human rater variance, omitting empirical human rater anomalies like rater halo, severity drift, or rater-item interactions.
The real scoring validation dataset was extremely small, comprising only six student essays.
Fairness distortions in the simulation were modeled as an additive shift of fixed magnitude rather than complex demographic-construct interactions.

From the article text (quote verified)

supports Global agreement metrics fail to capture demographic or contextual group disparities in automated grading. «ICC(A,1) and QWK were found to have high threshold exceedance performance even when systematic subgroup biases were introduced. This revealed that these metrics have serious limitations in detecting group-level inequalities.», p. 18, para 8

Model inference (gaps & contradictions)

[metric / conceptual blindspot]: Standard psychometric benchmarks for automated essay scoring (e.g., QWK >= 0.70) were developed for supervised regression/neural models rather than generative LLMs, which exhibit prompt-dependent midpoint compression and subtle demographic hallucinations that covariance metrics ignore.

#automated scoring #LLM as rater #inter-rater reliability #validity #subgroup fairness #Monte Carlo simulation #psychometrics #Bland-Altman