Can a System-One LLM Perform Knowledge Tracing When Few or No Learners Are Logged?
Authors: Unggi Lee, Haeun Park
Venue / Source: arXiv (2026)
A single-pass System-One LLM predicts student performance in new courses better than deep learning models trained on small cohorts.
Key takeaways
- Zero-shot single-pass LLMs can predict learner accuracy on new platforms without requiring student interaction logs.
- Providing a similar-learner accuracy statistic extends the cold-start advantage over deep knowledge tracing models up to cohort sizes of 64 students.
- System-Two reasoning chains increase latency and cost by orders of magnitude without improving knowledge tracing prediction accuracy.
- Supervised deep learning models require between 64 and 128 logged learners before consistently outperforming off-the-shelf System-One LLM predictions.
Executive Summary (TL;DR)
Authors' abstract (original)
Knowledge tracing (KT) models need many logged learners, so a new course or platform starts without a usable model. In LLM-based KT the LLM generates the answer, which we call System-Two; it is either fine-tuned on the target data or reasons and votes over ten samples, which is slow and gives coarse probabilities. We ask whether an off-the-shelf System-One LLM, which returns a probability for a typed question directly in a single pass, can perform KT when few or no learners are logged. On seven datasets, Jev without any data from the target platform reaches a mean AUC of .706, above the best of 28 deep KT models trained on 8 learners (.689) and above System-Two Thinking-KT on all seven datasets (.650) at about 1/100 of its API cost. Adding examples and a similar-learner statistic from the logged learners (JevKT) raises this to .722; JevKT stays significantly ahead of deep KT up to 16 learners and ahead on average up to 64, and supervised KT catches up between 64 and 128 learners. Among the readers we tested, the gain is specific to Jev, since three other LLMs queried with the byte-identical typed request through the official System-One adapter fall below it on all seven datasets, and reader swaps and contamination checks find no evidence that the input format or memorised data explain the gain. For new learners the advantage holds from their first interactions, whereas on unseen items with all learners logged, deep KT remains ahead.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Provides quantitative evidence on how generative LLMs model student proficiency and predict test response accuracy without custom retraining.
- Offers a benchmark method for evaluating item accuracy and difficulty under synthetic or zero-shot student prediction conditions.
- Demonstrates statistical calibration and item parameter interaction in zero-shot vs supervised student modeling.
Methodology and Dataset
Evaluated on 7 benchmark knowledge tracing datasets (ASSIST2009, NIPS34, XES3G5M, DBE-KT22, Algebra2005, Junyi, ASSIST17) comprising over 6 million interactions across thousands of items and knowledge components. Baselines included 28 pyKT deep knowledge tracing architectures (DKT, AKT, simpleKT, csKT, etc.), classical psychometric models (BKT, PFA), and System-Two LLM baselines (Thinking-KT, LOKT, CLST) using Qwen3.5-9B and Mistral-7B.
Why Worth Reading
This paper redefines the cold-start problem in educational data mining by showing that single-pass probabilistic language models outperform specialized deep learning architectures when data is scarce. For assessment specialism and platform design, it provides an immediately practical, calibration-ready method for student modeling that avoids costly retraining cycles during early course deployments.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Small sample: not found
- Unrepresentative sample: not found
- Single detector: not mentioned in the text
- Monolingual: not mentioned in the text
- Evaluated primarily on one closed, commercial System-One model (Jev), as open-source equivalents like Laya failed to achieve comparable accuracy.
- System-Two reasoning findings were tested on Qwen3.5-9B backbones and may not automatically generalize to much larger reasoning architectures.
- API-based inference introduces dependency on vendor-hosted model versions, though short-term re-querying demonstrated high consistency.
- supports Prompting standard language models with raw interaction histories yields near-chance accuracy without contextual population statistics. «Given only a raw interaction history, however, a prompted LLM predicts close to chance (Neshaei et al., 2024; Worden et al., 2026).», p. 3, para 2
- [method / conceptual mismatch]: Challenges the prevailing assumption in AIED that complex multi-step reasoning traces or fine-tuned deep learning architectures are necessary to achieve accurate personalized item performance predictions during cold-start regimes.