EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners
Authors: Unggi Lee, Sookbun Lee, Yeil Jeong, Eunjoo Lee, Minchul Shin, Hoilym Kwon
Venue / Source: arXiv (2026)
Standard AI tutor evaluations are often misleading because they rely on single-session metrics, failing to capture the long-term learning failure common in continuous agent-based tutoring.
Key takeaways
- Tutoring quality is not determined by the agent or base model alone but by the performance of the specific combination of both.
- Most current AI agents fail to sustain student learning gain beyond 5-10 days, suggesting a need for better long-horizon instructional design.
- There is a trade-off between immediate responsiveness and helpfulness; tutors that always answer requests immediately often score lower on pedagogy and helpfulness rubrics.
- Learning gain rankings are unstable across different student models, while helpfulness rubrics provide a more reliable anchor for evaluating AI tutors.
Executive Summary (TL;DR)
Authors' abstract (original)
Large language models (LLMs) power educational applications from tutoring to essay scoring, but each is a point solution to a single task, and only recently have these point solutions been integrated into agents operating over a learning management system (LMS). Yet tutoring is long-horizon, since a learner improves over days and weeks rather than in a single turn, and no benchmark evaluates an agent tutor across a sustained relationship. We introduce EduClaw-Bench, a benchmark that places an agent tutor in a continuous 30-day relationship with a simulated learner grounded in knowledge tracing (KT), whose knowledge-concept mastery, from a KT model trained on real-student data, drives its answers and is probed for learning gain across 55 scenarios. Each agent is scored on three primary axes (learning gain, responsiveness, and helpfulness) and two curriculum-design axes (Gagné and Rosenshine), with helpfulness and the curriculum axes judged by a cross-family panel of three LLM judges. Evaluating 10 agent adapters over three base-model tiers yields two findings that single-tier, single-session evaluation cannot reach. First, tutoring quality belongs to the base model and the agent harness together rather than either alone. Second, almost no combination sustains good tutoring over the full horizon. A calibration check ($\text{ECE}=0.049$) and a live-classroom field study confirm that the simulated learner and its measurements track reality. Our work is a step toward trustworthy AI tutors for future education.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Essential for understanding the validity and drift of automated pedagogical agents over time.
- Provides a concrete methodology for constructing AI-resilient benchmarks that account for long-term student engagement.
Methodology and Dataset
The benchmark evaluates 10 agent adapters across three tiers of base models (Solar-pro3, Codex-gpt5.5, Qwen3-4B-Thinking). The dataset comprises 55 scenarios per agent-tier combination, involving 11 learner personalities and 5 study schedules, totaling over 8,600 completed runs. Simulated learners are grounded in knowledge tracing (AKT) trained on 5,027 real student sequences from the XES3G5M dataset.
Why Worth Reading
This paper provides the first systematic evidence that popular AI tutor benchmarks are fundamentally flawed by their reliance on single-session metrics. It introduces a rigorous, reusable framework for longitudinal evaluation and proposes actionable safety gates for educators deploying AI agents, making it essential for anyone designing or vetting AI for long-term classroom integration.
Key Claims & Verified Quotes
No verified quotes available for this paper.
Limitations & Study Design Flags
- Simulated learners remain proxies and cannot fully replicate the complexity and unpredictability of real classroom students.
- The benchmark focuses on elementary mathematics and might not generalize to other subjects or older student demographics.
- Inter-agent learning gain variance is noisy at smaller scales, necessitating its use as a monitoring signal rather than a hard pass/fail gate.
- [time / methodological]: The paper identifies that static log benchmarks (like LongTutor) fail to capture the active learning loop, whereas EduClaw-Bench forces interaction with a dynamic learner model.