rct

Experimental Evidence on the Learning Impact of Generative AI

Authors: Zara Contractor, Germán Reyes
Venue / Source: arXiv (2026)

Using generative AI as a cognitive tutor fosters durable knowledge gains, whereas relying on it for automation offers only temporary benefits.

Key takeaways

Access to generative AI tools may enhance immediate academic performance and essay quality, with some benefits persisting in subsequent unassisted assessments.
The study employs a randomized controlled trial design, though results may be constrained by the use of a specific student population and specialized subject matter.
Learning outcomes vary significantly based on whether students utilize AI for cognitive augmentation or task automation, suggesting the pedagogical approach influences knowledge retention.
The research indicates that AI-assisted students prioritize information gathering and synthesis over rote writing, potentially shifting the nature of study activities.

Executive Summary (TL;DR)

In plain words In the paper's terms

This study tested how giving university students access to AI while learning a new topic affects their knowledge. Students with AI access did better on tests and essays immediately, and those benefits partly remained even when they were tested again a week later without any AI help. The results show that using AI as a tutor to help explain things is better for long-term learning than using it to write the work for you.

This randomized controlled trial investigates the impact of generative AI access on student learning productivity in a timed assignment. AI access increases immediate test performance and essay quality, with some gains persisting one week later in unassisted assessments. The findings suggest that learning outcomes are moderated by how students use AI, specifically distinguishing between cognitive 'augmentation' versus 'automation'.

Authors' abstract (original)

We study whether access to off-the-shelf generative AI changes how much students learn in a timed assignment. In a proctored, in-person randomized experiment, undergraduates study an unfamiliar topic and write an essay with or without AI, then complete unaided assessments immediately and one week later. AI access raises test scores, and a smaller gain persists a week later. Gains remain for students who use AI as a tutor ("augmentation") and fade for students who have AI write for them ("automation"). We document two mechanisms: students shift time from writing toward reading and searching for information, and report greater learning enjoyment.

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

5.0/10

Consensus Challenge

5.0/10

Topic Relevance

8.0/10

Relevance for Educators & Research

Provides a framework for assessing learning productivity vs task performance.
Offers a model for distinguishing between constructive tutoring effects and knowledge-eroding delegation.
Suggests validity threats to unproctored assessment where students might use 'automation' strategies.

Methodology and Dataset

N=211 undergraduate students at Middlebury College (Spring 2025). The study used a randomized controlled trial design across two in-person sessions one week apart. Students were randomized to AI-allowed (access to GPT-4o) or AI-forbidden conditions. Baselines included GPA, SAT scores, and pre-learning assessments.

Why Worth Reading

This paper provides some of the first rigorous causal evidence distinguishing between the 'tutoring' and 'delegation' effects of generative AI on learning. By measuring retention after a one-week delay without AI access, it directly addresses validity concerns about whether AI-assisted 'performance' reflects true knowledge acquisition or merely temporary output inflation.

Key Claims & Verified Quotes

AI access increases learning productivity in academic tasks.

Quote: “AI access improves learning outcomes, and a smaller gain remains after the tool is removed.” — 10.48550/arxiv.2607.08849, p. 3, para. 15 (exact)

Augmentation users retain knowledge gains, whereas automation users do not. overclaim

Quote: “the durability of learning gains depends on how students use AI” — 10.48550/arxiv.2607.08849, p. 23, para. 18 (exact)

Limitations & Study Design Flags

Unrepresentative sample: found
“The sample overrepresents women and first-year students” (p. 11)

Sample is limited to a single liberal arts college (Middlebury College) which may not be representative of diverse educational settings.
The study uses specific, technical, yet unfamiliar topics (blockchain, CRISPR, carbon capture), which may limit generalizability to other disciplines.
Detection of AI use relies on a commercial detector (Pangram 4) and LLM-based categorization of conversation logs.
Small sample size for subgroup analysis of AI 'automation' vs 'augmentation' behaviors.

From the article text (quote verified)

supports AI access improves learning outcomes in line with other AI-and-learning experiments. «Thus, our estimates are in line with the AI-and-learning experiments.», p. 17, para 22

Model inference (gaps & contradictions)

[method / Measurement]: The paper distinguishes between 'augmentation' and 'automation' through LLM-coded conversation logs, but this remains a self-selected behavior rather than a randomized treatment assignment within the AI group.

#higher_education #assessment_validity #generative_ai #rct #learning_analytics #item_response_theory_adjacent