psychometric_modeling

From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMs

Authors: Hanmeng Liu, Shichao Weng, Xiulai Liu, Zhicai Zhang, Anli Yan, Xiaozhang Liu
Venue / Source: arXiv (2026)

Current multiple-choice tests struggle to measure real AI reasoning because AI models can memorize answers or exploit simple shortcuts.

Executive Summary (TL;DR)

In plain words In the paper's terms

Current multiple-choice tests struggle to measure real AI reasoning because AI models can memorize answers or exploit simple shortcuts. This study introduces a method that automatically converts traditional multiple-choice options into complex logical combinations that models must evaluate under strict rules. When tested on this upgraded format, top AI models experienced drastic performance drops because they struggled to evaluate multiple logical statements simultaneously.

Multiple-choice reasoning benchmarks suffer from benchmark saturation and static evaluation failures due to surface pattern memorization. The authors propose LogiHard, a formal framework that deterministically transforms 0-order selection into 2-order propositional logical judgment while incorporating Item Response Theory for adaptive testing. Across twelve frontier large language models, evaluation on LogiHard-2k reveals a 31% to 56% accuracy degradation caused by multi-select failure and early exit bias.

Authors' abstract (original)

Multiple-choice reasoning benchmarks face dual challenges: rapid saturation from advancing models and data contamination that undermines static evaluations. Ad-hoc hardening methods (paraphrasing, perturbation) attempt to increase difficulty but sacrifice logical validity for surface complexity, falling short to challenge advanced reasoning models. We present LogiHard, a formal framework that deterministically transforms 0-order selection into 2-order logical judgment, which significantly increases the thinking overhead and reasoning steps. The framework integrates Item Response Theory (IRT) for computerized adaptive testing (CAT), enabling precise difficulty control with fewer questions than static benchmarks. We instantiate LogiHard-2k, a logical reasoning dataset constructed by cognitively ranking high-stakes examination questions via 9-dimensional analysis of model thinking traces, followed by combinatorial transformation of high-difficulty items. Evaluation across twelve state-of-the-art models reveals an accuracy degradation ranging from 31% to 56% on combinatorially hardened questions. LLMs suffer from the multi-select failure and early exit bias, which are not shared by human testees. Zero-shot transfer to MMLU demonstrates 47% accuracy degradation (89.84% to 42.86%), confirming applicability across domains with provable validity preservation. The consistent aggregate degeneration is domain-agnostic and stems not from knowledge deficits but from a combinatorial reasoning gap, reflecting a training-induced completeness-verification deficit.

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

8.0/10

Consensus Challenge

8.0/10

Topic Relevance

8.0/10

Methodology and Dataset

Sample size: N = 2,000 high-stakes examination items (1,461 atomic LogiHard-Base items and 539 combinatorially transformed LogiHard-C items derived from a 6,235-item corpus), 128 MMLU transfer items, and n = 30 graduate-level human evaluators. Tested LLMs: 12 frontier models including GLM-5, GLM-4.7, GPT-5.4, o3, Claude-Opus-4.6, DeepSeek-R1, DeepSeek-V3.2, DeepSeek-V4-Pro, Gemini-3.1-pro, Kimi-k2.5, Qwen3.5-Plus, and Qwen3.6-plus. Baselines: raw corpus original MCQs, None-of-the-Above (NOTA) insertion, and option shuffling.

Why Worth Reading

This paper is essential for educational assessment researchers studying AI validity because it demonstrates how standard multiple-choice questions can be systematically hardened against AI shortcuts and memorization. By elevating 0-order option selection to 2-order logical constraint satisfaction and integrating psychometric IRT-CAT modeling, it provides a blueprint for constructing adaptive, AI-resistant assessments while offering quantitative diagnostics of LLM reasoning limitations.

Key Claims & Verified Quotes

LogiHard deterministically transforms 0-order option selection into 2-order logical judgment to increase reasoning complexity.

Quote: “We present LogiHard, a formal framework that deterministically transforms 0-order selection into 2-order logical judgment, which significantly increases the thinking overhead and reasoning steps.” — 10.48550/arxiv.2605.07268, p. 1, para. 5 (exact)

Hardening questions via propositional logic causes severe accuracy drops across twelve state-of-the-art language models.

Quote: “Evaluation across twelve state-of-the-art models reveals an accuracy degradation ranging from 31% to 56% on combinatorially hardened questions.” — 10.48550/arxiv.2605.07268, p. 1, para. 5 (exact)

Computerized adaptive testing using Item Response Theory reduces required test length while preserving measurement precision.

Quote: “This protocol achieves measurement precision equivalent to static 60-item tests using only 15–25 adaptively selected questions per subset, while precisely targeting the difficulty zone appropriate to each model’s capability.” — 10.48550/arxiv.2605.07268, p. 6, para. 1 (exact)

Limitations & Study Design Flags

Small sample: not found
Unrepresentative sample: not mentioned in the text
Single detector: not mentioned in the text
No adversarial testing: not mentioned in the text