MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring
Woojin Lee, Yoonkyu Woo, Jinah Kim, Hyeok-Min Gwon, Jin‐Xia Huang · arXiv (Cornell University) (2026) · doi:10.48550/arxiv.2610.31679 · gemini-3.6-flash
TL;DR
Evaluating how effectively AI math tutors remediate student misconceptions depends heavily on the chosen grading criteria. When multi-turn tutoring dialogues are graded with strict exact-answer matching, AI tutors appear to succeed only 42% of the time, compared to 81% when evaluated by human-calibrated AI judges accepting valid conceptual reasoning. Furthermore, feeding AI tutors detailed misconception diagnoses provides no measurable learning benefit over direct tutoring.
Method and sample: The study evaluates five open-weight models (Qwen-2.5-7B, Mistral-7B, Llama-3.1-8B, Phi-4 14B, Qwen3.5-9B) plus two proprietary models (Claude Haiku 4.5, Gemini 3.1 Flash Lite). Detection is tested on BEA 2025 (N=300), MathDial (N=2,262), and Bridge (N=419). Diagnosis uses the Math Misconceptions Benchmark (N=220, 55 middle-school algebra misconceptions). Remediation is tested across 106 instantiated scenarios using GPT-5-mini simulated students across 2,250 dialogues in Exp. 3 and 3,180 dialogues in Exp. 4. A human calibration study evaluates 200 quality-stratified dialogues (100 clean, 100 borderline) using three blinded human raters.
#Validity Arguments#LLMs as Raters#LLMs as Test Takers#Assessment Redesign#Mathematics Education#Automated Scoring#Inter-Rater Reliability