quasi_experiment

Faster Completion, Less Learning: Generative AI Reduced Study Time on Math Problems and the Knowledge They Build

Authors: Sina Rismanchian, Hasan Uzun, Jeffrey Matayoshi, Eric Cosyn, Eyad Kurd-Misto
Venue / Source: arXiv (2026)

This study examined how generative AI tools have changed the way millions of students complete math problems.

Executive Summary (TL;DR)

Wording

This study examined how generative AI tools have changed the way millions of students complete math problems. The researchers found that students are finishing text-based problems significantly faster, which is associated with a notable decrease in their ability to retain information. This pattern suggests that students are outsourcing their thinking to AI instead of doing the work themselves, leading to weaker learning outcomes.

This study analyzes 3.2 million learning interactions and 12.2 million assessment response times to investigate the impact of generative AI on student behavior and learning. Using a quasi-experimental design based on variation in AI-susceptibility across problem types, the authors find significant declines in learning time for word problems in post-ChatGPT quarters. Results demonstrate that this behavioral shift correlates with a 25% cumulative reduction in durable knowledge retention, evidenced by performance drops on proctored assessments.

Authors' abstract (original)

How much have students' ordinary learning processes shifted in response to generative AI, and how does that affect their durable learning outcomes? Self-report surveys show little change, while small-scale behavioral studies report widespread AI use without the scale or duration to measure learning consequences. We address both questions using a ten-year panel of $3.2$ million ALEKS learning interactions for investigating time-on-task, complemented by ALEKS PPL placement-assessment data for examining proctoring and learning outcomes, with a quasi-experimental design exploiting variation in tasks that are more susceptible to AI (text-based word problems) and less susceptible to AI (interactive graph-based problems). Learning time on AI-susceptible problems declines $2.8\%$ per quarter among college students after ChatGPT's release, cumulating to $26.9\%$ over eleven quarters; high-schoolers show $31.3\%$, middle-schoolers $9.0\%$, and Grade 5 students no detectable change. Among college students, the post-ChatGPT divergence vanishes entirely under proctoring, ruling out broad efficiency gains as the likely explanation. Logistic fixed-effects models on randomly assigned proctored retention items yield a $25\%$ cumulative decline in odds of correct response; the same estimator on non-proctored assessment produces a large opposite-signed increase -- inconsistent with any platform, cohort, or curriculum explanation. These results are among the first large-scale behavioral and outcome evidence that generative AI has altered how students study and the knowledge they build -- the population-level indicator of \emph{cognitive surrender}, with direct implications for educational research, assessment governance, and AI policy.

Quality & Relevance Evaluation Breakdown

Novelty
8.0/10
Methodological Rigor
8.0/10
Engineering Countermeasures
5.0/10
Consensus Challenge
5.0/10
Topic Relevance
8.0/10

Methodology and Dataset

The study analyzes a ten-year panel (2015-2025) of 3.2 million ALEKS learning interactions and 12.2 million ALEKS PPL assessment response-time observations. The sample spans Grade 5 through College Algebra. The methodology utilizes a quasi-experimental design comparing 'AI-susceptible' (text-based) and 'AI-resistant' (graphical/interactive) problem types, with falsification tests conducted via proctored vs. unproctored assessment conditions.

Why Worth Reading

It provides large-scale, longitudinal, behavioral evidence of 'cognitive surrender' in an authentic educational platform, linking AI adoption not just to shortcuts in homework, but to a measurable decline in durable student knowledge.

Key Claims & Verified Quotes

Learning time on AI-susceptible math problems declined significantly across college and high school populations following the release of ChatGPT.
Quote: “Learning time on AI-susceptible problems declines 2.8% per quarter among college students after ChatGPT’s release, cumulating to 26.9% over eleven quarters; high-schoolers show 31.3%, middleschoolers 9.0%” — 10.48550/arxiv.2605.21629, p. 1, para. 21 (exact)
Reduced engagement with AI-susceptible problems during learning is associated with a cumulative decline in retention performance on proctored assessments.
Quote: “Logistic fixed-effects models on randomly assigned proctored retention items yield a 25% cumulative decline in odds of correct response” — 10.48550/arxiv.2605.21629, p. 1, para. 21 (exact)

Limitations & Study Design Flags