From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMs
Hanmeng Liu, Shichao Weng, Xiulai Liu, Zhicai Zhang, Anli Yan, Xiaozhang Liu · arXiv (2026) · doi:10.48550/arxiv.2605.07268 · gemini-3.6-flash
TL;DR
Current multiple-choice tests struggle to measure real AI reasoning because AI models can memorize answers or exploit simple shortcuts. This study introduces a method that automatically converts traditional multiple-choice options into complex logical combinations that models must evaluate under strict rules. When tested on this upgraded format, top AI models experienced drastic performance drops because they struggled to evaluate multiple logical statements simultaneously.
Method and sample: Sample size: N = 2,000 high-stakes examination items (1,461 atomic LogiHard-Base items and 539 combinatorially transformed LogiHard-C items derived from a 6,235-item corpus), 128 MMLU transfer items, and n = 30 graduate-level human evaluators. Tested LLMs: 12 frontier models including GLM-5, GLM-4.7, GPT-5.4, o3, Claude-Opus-4.6, DeepSeek-R1, DeepSeek-V3.2, DeepSeek-V4-Pro, Gemini-3.1-pro, Kimi-k2.5, Qwen3.5-Plus, and Qwen3.6-plus. Baselines: raw corpus original MCQs, None-of-the-Above (NOTA) insertion, and option shuffling.