psychometric_modeling

QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks

Authors: Pranav Gupta
Venue / Source: arXiv (2026)

Aggregate LLM performance rankings frequently mask severe competence gaps in specialized tasks like circuit routing or debugging.

Key takeaways

LLM capability in specialized domains like quantum computing is not unidimensional; aggregate scores may hide zero-percent success rates on specific sub-tasks.
Item Response Theory (IRT) provides a rigorous way to validate that benchmark difficulty scales appropriately and that ability measurements are reliable.
Hardware routing in quantum circuits acts as a universal bottleneck, with even top-performing models struggling to maintain semantics while inserting connectivity gates.
Model performance on quantum tasks is sensitive to prompt engineering, though top-tier models maintain relatively stable rankings across conditions.
The 59% performance drop from textbook-level to open-research-level problems underscores that benchmark difficulty scaling is critical for realistic assessment.

Executive Summary (TL;DR)

In plain words In the paper's terms

Researchers created a new set of 11 challenging quantum computing tests to see how different AI models handle complex, practical work. They found that an AI's total score often hides the fact that it might be excellent at one type of task but completely fail at another. Their results show that these models are not equally skilled across different areas, which makes simple overall rankings misleading for educators.

This study introduces QC-Stark, a 11-task quantum computing benchmark evaluated across 10 LLMs to analyze capability dissociations and measurement reliability. Using a 2-parameter Item Response Theory (IRT) model, the authors quantify task difficulty and model ability, revealing that aggregate rankings often obscure substantial per-task performance variations. The research finds that model competence is not unidimensional, with significant rank inversions occurring between overall scores and specific tasks like circuit routing or debugging.

Authors' abstract (original)

We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debugging, compilation, error correction, and simulation. Across 2,750 evaluations (10 models $\times$ 11 tasks x 5 difficulty levels x 5 seeds), we find that overall rankings mask substantial per-task variation. The Spearman correlation between overall and per-task rankings is statistically insignificant for 4 out of the 11 tasks included in this benchmark. A 2-parameter Item Response Theory (IRT) model validates measurement quality, and prompt sensitivity analysis confirms ranking robustness across prompt conditions. All tasks are auto-verifiable via execution, thus not requiring any manual evaluation. We make the code and data publicly available on Huggingface.

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

5.0/10

Consensus Challenge

5.0/10

Topic Relevance

8.0/10

Relevance for Educators & Research

Provides a methodology for detecting 'capability dissociations' which can be used to analyze how LLMs perform on different types of academic assessments.
Demonstrates the utility of 2PL IRT models in verifying the quality of educational assessment items.
The findings on how task difficulty levels interact with LLM performance provide a basis for designing assessment tasks that are more resilient to AI 'shortcuts'.

Methodology and Dataset

The dataset consists of 2,750 evaluations (10 models × 11 tasks × 5 seeds × 5 difficulty levels). The evaluated models include: Claude Sonnet 5, Gemini 3.5 Flash, o4-mini, GPT-5.4, Gemma-4 31B, GPT-4.1 mini, Claude Opus 4.1, Gemini Flash Lite, Mistral Large 3, and LLaMA-3.3 70B.

Why Worth Reading

It provides a clear framework for applying Item Response Theory to LLM benchmarks, offering a robust method to measure 'ability' versus 'difficulty' that is directly transferable to assessing AI-resistant student examinations.

Key Claims & Verified Quotes

A single aggregate score is insufficient in guiding model selection for quantum computing tasks due to performance inversions.

Quote: “These inversions demonstrate that a single aggregate score is insufficient in guiding model selection for quantum computing tasks” — 10.48550/arxiv.2609.35581, p. 3, para. 9 (exact)

Limitations & Study Design Flags

Small sample: not found
Unrepresentative sample: not mentioned in the text
Single detector: not mentioned in the text

The study uses Python/Qiskit 2.x, which may conflate quantum domain reasoning with framework-specific API proficiency.
Evaluation is based on model availability, potentially biasing results towards specific developer families (e.g., Anthropic).
Some evaluations (11 records) remained truncated at hard API token caps, and 46 evaluations timed out during verification.
A small number of difficulty-level/model cells had to be excluded from prompt sensitivity analysis due to persistent API errors.

#item_response_theory #benchmarking #AI_assessment_resilience #quantitative_evaluation #capability_dissociation