psychometric_modeling

Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation

Authors: Nishant Balepur, Paiheng Xu, Wei Ai, Eunsol Choi, Rachel Rudinger, Jordan Boyd-Graber
Venue / Source: arXiv (2026)

Standard multiple-choice scoring only measures accuracy, but adopting education-inspired schemes reveals hidden model behaviors like abstention, self-correction, and eliminative reasoning.

Key takeaways

Alternative scoring schemes (e.g., elimination, confidence calibration, answer-until-correct) significantly shift model rankings compared to standard accuracy.
The 'Answer Until Correct' (AUC) scoring scheme aligns best with user preferences observed in the LLM Arena.
Different scoring schemes surface specific model behaviors such as abstention rates, confidence levels, and willingness to revise answers, which standard accuracy fails to capture.
Stronger LLMs generally exhibit more logical consistency across different scoring schemes than smaller, weaker models.

Executive Summary (TL;DR)

In plain words In the paper's terms

Current AI tests usually just count how many multiple-choice questions a model gets right, which often encourages guessing. By using alternative scoring methods from traditional education, this study shows that we can better measure a model's true reasoning, its ability to fix mistakes, and how much it actually understands. These new metrics better predict which AI models people will prefer to use in real-world settings.

This paper evaluates six education-grounded scoring schemes for multiple-choice question answering (MCQA) to assess abilities beyond simple accuracy. The study finds that alternative schemes shift LLM rankings and that an 'Answer Until Correct' (AUC) approach best correlates with human preferences in the LLM Arena. These findings demonstrate that scoring scheme design is a critical, often overlooked lever for better evaluating LLM capabilities and logical consistency.

Authors' abstract (original)

Multiple-choice question answering (MCQA) benchmarks in NLP use number-right scoring (accuracy), but in educational testing, the scoring scheme, the combination of the response mode models follow and the rule for grading responses, is a key design choice that dictates which abilities to reward. We examine how alternatives to number right change what MCQA measures with six education-inspired schemes that assess abilities beyond accuracy: distractor elimination, abstention, confidence calibration, and self-correction. On LLM benchmarks, these schemes: 1) shift rankings of 31 LLMs beyond rephrased number right prompts; 2) better predict the LLMs users prefer in LLM Arena; and 3) reveal distinct model capabilities, like that GPT-5 rarely abstains and readily self-corrects, while weaker open-weight models often abstain and hesitate to eliminate choices. Given the benefits of alternative scoring schemes, we discuss ways to extend them to tasks beyond MCQA.

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

8.0/10

Consensus Challenge

5.0/10

Topic Relevance

8.0/10

Relevance for Educators & Research

Provides a framework for 'AI-resistant' assessment design by suggesting that shifting response modes can reveal whether a model has true reasoning or is merely guessing.
Offers clear methods for implementing non-standard scoring (elimination, confidence-based marking) which can be used to detect AI behavior shifts.

Methodology and Dataset

The study evaluates 31 frontier LLMs (including models from Gemini, GPT, Claude, Llama, and Qwen families) across 1,000 MCQ examples sampled from ARC (grade-school science), MMLU (college knowledge), and SuperGPQA (graduate-level expertise) benchmarks.

Why Worth Reading

This paper provides a rare, rigorous look at the scoring layer of assessment, offering educators and developers a concrete way to move beyond 'number right' accuracy metrics toward more robust, behavior-oriented assessment frameworks.

Key Claims & Verified Quotes

Alternative MCQA scoring schemes better capture distinct model capabilities such as confidence calibration and self-correction than standard accuracy.

Quote: “These schemes surface student abilities that number right scoring cannot distinguish” — 10.48550/arxiv.2608.29887, p. 1, para. 10 (exact)

Answer Until Correct (AUC) scoring more reliably predicts human-preferred LLM rankings than traditional accuracy.

Quote: “AUC best agrees with LLM Arena (Figure 2)—more than NR—so AUC can better predict LLMs users prefer pre-deployment.” — 10.48550/arxiv.2608.29887, p. 3, para. 18 (exact)

Limitations & Study Design Flags

LLM performance is sensitive to specific prompt formatting, potentially confounding results.
The Answer Until Correct (AUC) method is significantly more computationally expensive due to multi-turn interactions.
The study assumes standard education-inspired schemes are effective without validating their psychometric properties specifically for LLM-based testing.

From the article text (quote verified)

supports Standard number-right scoring persists due to simplicity despite concerns that it rewards guessing. «Despite longstanding concerns that this rewards guessing (Du et al., 2023) and poorly reflects abilities users value (Saxon et al., 2024), it persists due to its simplicity.», p. 1, para 9
supports Users prefer LLMs that adapt to feedback, which AUC effectively models. «This matches Shi et al. (2024) and Liu et al. (2025) who show users prefer LLMs that adapt well to feedback, which AUC captures by making LLMs self-correct», p. 3, para 18

Model inference (gaps & contradictions)

[method / psychometric gap]: While adopting schemes from education, the authors do not explicitly confirm if these schemes maintain the same construct validity when applied to LLMs compared to human test-takers.

#psychometrics #assessment_design #evaluation_metrics #LLM_reasoning #logical_consistency