psychometric_modeling

The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost

Authors: Scott Frohn
Venue / Source: arXiv (2026)

Repeatedly sampling the same model does not improve automated scoring accuracy, but choosing low-cost models without reasoning drastically cuts costs.

Key takeaways

Intra-model majority voting across multiple runs fails to improve scoring accuracy because errors stem from systematic rubric misinterpretation rather than random output noise.
Higher reasoning effort slightly improves scoring accuracy on average, but the benefit is inconsistent across model families and plateaus quickly.
Sampling at temperature 1 outperforms deterministic greedy decoding at temperature 0 for smaller models without requiring ensembling.
Low-cost compact models operating without chain-of-thought reasoning offer an optimal cost-accuracy frontier for high-volume automated scoring deployments.

Executive Summary (TL;DR)

In plain words In the paper's terms

This study tested whether having AI grade a student's open-ended math explanation multiple times or think more deeply before scoring improves accuracy. Grading multiple times with the same AI did not improve scoring accuracy at all compared to a single evaluation. Low-cost AI models with no extra deliberation achieved nearly the same accuracy as expensive models at a fraction of the cost.

Automated scoring of constructed responses via large language models often incurs substantial inference cost, but the accuracy benefits of intra-model self-consistency and reasoning effort remain unclear. Evaluating 900 student conversations on high school mathematics items across frontier and low-cost OpenAI and Gemini models, this study tested temperature sampling, ensemble sizes (j = 1 to 7), and reasoning levels against human majority-vote ground truth. Increasing ensemble size produced no significant accuracy gains (p = .379), whereas higher reasoning effort yielded a significant positive linear trend (p < .001) dominated by frontier Gemini 3.1 Pro.

Authors' abstract (original)

Strategic model selection and reasoning settings are more effective than ensembling for optimizing automated scoring with large language models (LLMs). We examined self-consistency (intra-model majority voting) and reasoning effort for scoring conversation-based assessment items in high school mathematics, evaluating 900 student conversations against human-scored ground truths using frontier and low-cost models from OpenAI and Google. Temperature sampling significantly improved accuracy over deterministic calls, but increasing ensemble size (j = 1 to 7) produced no significant gains. Higher reasoning effort showed a significant positive linear trend with scoring accuracy, though the benefit varied by model family. An efficiency frontier analysis identified Gemini 3.1 Pro Preview at low reasoning as the most accurate but costly configuration; GPT-5.4 Nano and Mini with no reasoning offered the best cost-performance balance.

Quality & Relevance Evaluation Breakdown

Novelty

6.0/10

Methodological Rigor

7.5/10

Engineering Countermeasures

7.0/10

Consensus Challenge

7.0/10

Topic Relevance

7.5/10

Relevance for Educators & Research

Evaluates LLMs as raters on interactive dialogue-based constructed response mathematics assessments.
Quantifies the psychometric stability and rater drift/bias of automated scoring systems across temperature and reasoning parameters.
Directly informs operational assessment engineering tradeoffs between rater validity, scoring latency, and monetary cost.

Methodology and Dataset

Evaluated 900 student conversations across three high school mathematics Explain Your Thinking items (600 Algebra I, 300 Geometry) scored on checklist rubrics with 2–3 dichotomous criteria (2,100 ground-truth ratings established by majority vote of three trained human raters). Evaluated models included OpenAI GPT-5.4, GPT-5.4-Mini, GPT-5.4-Nano, and Google Gemini 3.1 Pro Preview and Gemini 3 Flash Preview. Experimental conditions manipulated temperature (0 vs 1), ensemble size (j = 1, 3, 5, 7), and reasoning level (none/minimal, low, medium, high), analyzed via generalized linear mixed models (GLMM) and Pareto efficiency frontiers.

Why Worth Reading

It provides rigorous empirical evidence debunking the assumption that self-consistency majority voting benefits automated scoring, while offering an actionable cost-performance Pareto analysis for operational assessment systems.

Key Claims & Verified Quotes

Higher reasoning effort linearly improves scoring accuracy overall, but the effect departs from strict monotonicity.

Quote: “Higher reasoning effort significantly improved scoring accuracy (linear contrast: b = 0.145, z = 4.06, p < .001).” — 10.48550/arxiv.2604.26954, p. 7, para. 2 (exact)

Limitations & Study Design Flags

Small sample: not found

Restricted exclusively to dichotomously scored checklist rubrics in high school mathematics, limiting generalization to polytomous essays or non-STEM domains.
Evaluated only intra-model self-consistency, leaving multi-model heterogeneous ensembles unaddressed.
Cost estimates rely on March 2026 commercial API list prices without accounting for batch API discounts or prompt caching.

Model inference (gaps & contradictions)

[method / divergence]: Benchmarks in general NLP show self-consistency yielding 4–18% accuracy boosts, but rubric-based assessment tasks do not benefit because LLMs exhibit systematic rubric bias rather than stochastic drift.
[metric / conceptual]: High intra-model reliability (Fleiss' kappa > 0.88) was observed, yet human-model agreement remained capped at Cohen's kappa ≈ 0.79, indicating reliable misclassification.

#automated scoring #large language models #self-consistency #reasoning effort #interrater reliability #cost efficiency