SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models
Authors: Berke Arda, Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mubashara Akhtar, Mrinmaya Sachan
Venue / Source: arXiv (2026)
Cognitive diagnosis models reveal fine-grained LLM skill profiles, enabling 4x cheaper query routing while outperforming scalar benchmark scores in stability.
Key takeaways
- Average benchmark scores obscure fine-grained capability differences between language models with identical scalar performance.
- Neural Cognitive Diagnosis Models anchor coordinate systems to named skills, eliminating the non-identifiability of multidimensional IRT.
- Skill-based LLM routing cuts inference costs by up to 75% while maintaining accuracy equal to the strongest single model.
- Computerized adaptive testing profiles unseen language models with 20 targeted items as accurately as 100 random items.
Executive Summary (TL;DR)
Authors' abstract (original)
Current evaluation of LLMs on benchmarks typically aggregates performance over individual examples into a single scalar score. This approach is overly coarse and ignores the diverse scope of skills covered by benchmark datasets. To address this limitation and better exploit benchmark data, we propose SkillEval, a new evaluation framework that performs cross-model, skill-level analysis. The framework integrates advances from two areas: (1) it leverages the meta-cognitive capabilities of LLMs to automatically discover interpretable skills from questions, and (2) it applies cognitive diagnosis models (CDMs) from psychometrics to estimate per-skill abilities of LLMs. We conduct experiments on 3,811 LLMs evaluated on 9,523 items across five benchmarks spanning diverse domains (MATH, BBH, GPQA, MuSR, IFEval). Compared to the traditional average-based evaluation paradigm, SkillEval produces reliable, more calibrated and informative yet concise model profiles that more accurately predict model behavior on unseen items. We also demonstrate the utility of SkillEval in two downstream tasks: (1) LLM routing, which dynamically assigns each query to a capable but efficient model, reducing inference cost by over 60% while achieving improved performance, and (2) Computerized Adaptive Testing (CAT), where an unseen model can be efficiently evaluated using a trained CDM. Furthermore, we show that these skill profiles provide useful insights, for example, how benchmarks share skills and how instruction-tuned models differ from their base models.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Provides a concrete methodological blueprint for applying psychometric Cognitive Diagnosis Models (CDMs) and Q-matrix construction to generative AI evaluation.
- Demonstrates how Computerized Adaptive Testing (CAT) principles reduce item sampling burden while maintaining diagnostic validity.
- Offers mathematical formulations for text-conditioned item parameter estimation suitable for dynamic assessment banks.
Methodology and Dataset
The dataset contains 36 million model-item response records covering M=3,811 LLMs and N=9,523 items across five benchmarks (MATH, BBH, GPQA, MuSR, IFEval). GPT-4o-mini extracts 13,276 unique skill strings (mean 2.9 per item), which are embedded via SBERT (all-mpnet-base-v2) and grouped via hierarchical agglomerative clustering into K=100 skill clusters to construct a binary Q-matrix. A text-conditioned Neural CDM (NCDM) learns item difficulty vectors and discrimination parameters from text embeddings while constraining model skill mastery to [0,1]. Baselines include 2PL IRT, multidimensional IRTNet (d=232, 512, 1024), KNN item retrieval, and single top-performing LLMs.
Why Worth Reading
This paper bridges psychometric cognitive diagnosis modeling and large language model evaluation. It directly addresses the scalar score validity crisis in LLM benchmarks by offering a robust, interpretable, and reproducible multi-dimensional profiling alternative. Psychometricians and assessment researchers will find the application of CD-CAT, Q-matrix extraction, and neural CDMs highly transferable to automated human assessment and LLM evaluation design.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Small sample: not found
- Unrepresentative sample: not mentioned in the text
- Single detector: not mentioned in the text
- No adversarial testing: not mentioned in the text
- Monolingual: not mentioned in the text
- Item difficulty and discrimination parameters learned from text embeddings are less precise for predicting individual item responses than standard unconstrained IRT parameters.
- Q-matrix construction errors propagate directly into item response and ability profile predictions.
- Observational skill mastery comparisons between base and instruction-tuned models do not establish direct causal effects of instruction tuning.
- supports Multidimensional IRT models suffer from poor interpretability and non-identifiability in repeated runs. «However, these multidimensional models are known to suffer from poor interpretability and non-identifiability, making them less suitable for reliable evaluation.», p. 2, para 2
- [method / conceptual difference]: Contradicts standard scalar IRT and benchmark aggregate scores by demonstrating that scalar latent ability creates capability collapse across distinct domain sub-skills.