SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models
Berke Arda, Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mubashara Akhtar, Mrinmaya Sachan · arXiv (2026) · doi:10.48550/arxiv.2610.28767 · gemini-3.6-flash
TL;DR
Evaluating AI models with a single average score hides specific strengths and weaknesses across different tasks. This paper introduces SKILLEVAL, a method that breaks down test questions into specific skills and uses educational measurement models to rate AI models on each skill. This skill profile remains stable across re-testing, helps automatically route user queries to cheaper AI models without losing accuracy, and tests new AI models faster.
Method and sample: The dataset contains 36 million model-item response records covering M=3,811 LLMs and N=9,523 items across five benchmarks (MATH, BBH, GPQA, MuSR, IFEval). GPT-4o-mini extracts 13,276 unique skill strings (mean 2.9 per item), which are embedded via SBERT (all-mpnet-base-v2) and grouped via hierarchical agglomerative clustering into K=100 skill clusters to construct a binary Q-matrix. A text-conditioned Neural CDM (NCDM) learns item difficulty vectors and discrimination parameters from text embeddings while constraining model skill mastery to [0,1]. Baselines include 2PL IRT, multidimensional IRTNet (d=232, 512, 1024), KNN item retrieval, and single top-performing LLMs.
#Cognitive Diagnosis Models#Psychometrics#LLM Evaluation#Computerized Adaptive Testing#Q-Matrix#Item Response Theory#Model Routing