quasi_experiment

When the Scaffold Stays On: AI, Practice Style, and Screening in Elite Skill Formation

Authors: Song Yao
Venue / Source: arXiv (2026)

Generative AI can be used to either replace learning or speed it up, and these two behaviors are hard to distinguish in day-to-day practice.

Executive Summary (TL;DR)

In plain words In the paper's terms

Generative AI can be used to either replace learning or speed it up, and these two behaviors are hard to distinguish in day-to-day practice. The study shows that in open online programming practice, AI-like behavior often signals skill erosion, but among top-tier contestants who have passed strict, AI-prohibited tests, the same behavior actually predicts higher performance. This means that designing assessments to test unaided skill can effectively identify and encourage students who use AI as a helpful tool rather than a crutch.

This study analyzes skill formation in competitive programming by comparing practice behavior and unaided contest outcomes across AI-prohibited and open environments. Researchers find that while AI-style practice patterns emerged across all cohorts, they predict skill erosion only among un-screened users in the open pool. Inside AI-prohibited settings, however, shifts toward this same practice style consistently predict higher unaided contest performance, suggesting complement-style usage among elite-screened learners.

Authors' abstract (original)

Generative AI raises short-term productivity by completing tasks learners would otherwise practice on their own. Whether this exchange erodes frontier skill depends on the mode of use: substitute-users let AI stand in for practice and fail to develop skills, while complement-users use AI to learn faster. The modes look alike in AI-aided output, so organizations screening on that output cannot tell them apart. We ask whether the AI-prohibited evaluation gates organizations already operate can separate the modes. In elite competitive programming, ICPC and IOI contests prohibit AI under in-person proctoring, with qualification-round entry, whereas Codeforces (CF) practice and contests are unproctored and open to all. From CF practice histories we build an AI-prompt signature consistent with AI usage, more first-attempt acceptances, fewer attempts and debugging retries. CF practice has shifted toward this signature across entry cohorts spanning two AI rollouts. On CF, a stronger signature predicts smaller rating gains for users with no ICPC-IOI affiliation, but not for those who qualified. Inside the AI-prohibited ICPC environment, AI-era entrants show no skill erosion, and shifts toward AI-style practice predict higher non-AI-aided scores. One screening mechanism fits both: where the modes mix, a stronger signature flags substitute-users; among those who qualified, a strengthening signature marks adoption of the complement mode. The message is constructive: AI-style practice is compatible with frontier skill; the erosion risk links to the substitute mode; and separating the modes is a design question for the exams organizations regularly administer, from medical and legal boards to professional certification.

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

5.0/10

Consensus Challenge

5.0/10

Topic Relevance

8.0/10

Relevance for Educators & Research

Provides a framework for distinguishing between substitution and complementarity in learning technologies.
Demonstrates the efficacy of AI-prohibited gates as a screening mechanism for verifying unaided skill.
Offers a quantitative methodology for detecting shifts in practice styles ('AI-prompt signature') using platform data.

Methodology and Dataset

The sample includes 10,419 Codeforces (CF) users observed from 2018 to 2026. This population is divided into a rostered subpopulation (2,480 users) linked to AI-prohibited ICPC/IOI contest records and a non-rostered expansion of 7,939 users. Models tested include difference-in-differences on practice signatures and cross-subpopulation performance regressions.

Why Worth Reading

This paper provides rare, multi-year empirical evidence on how AI integration affects long-term human capital formation. It demonstrates that the impact of AI on learning depends on the assessment environment, offering a concrete model for how educators can maintain assessment validity by forcing the 'scaffold' of AI to be removed during high-stakes credentialing.

Key Claims & Verified Quotes

AI use during learning can be classified into substitute and complement modes, which affect skill formation differently.

Quote: “Whether this exchange erodes frontier skill depends on the mode of use: substitute-users let AI stand in for practice and fail to develop skills, while complement-users use AI to learn faster.” — 10.48550/arxiv.2606.06253, p. 1, para. 3 (exact)

AI-prohibited assessment gates can effectively filter out substitute-mode users.

Quote: “Once the gate has screened substitute-users out, the remaining AI-style practice tracks complement use.” — 10.48550/arxiv.2606.06253, p. 4, para. 28 (exact)

Limitations & Study Design Flags

The AI-prompt signature is a proxy based on behavioral patterns and does not constitute a direct, verified measure of AI tool usage.
The cross-cohort practice shift is descriptive and does not show clear alignment with the exact timing of AI tool releases.
The analysis of roster-linked vs. non-rostered populations relies on existing contest participation as a proxy for screening status, which may not capture all selection effects.

Model inference (gaps & contradictions)

[method / Measurement]: The study relies on behavioral proxies for AI usage rather than direct access or interaction logs, limiting the causal strength of the practice-mode interpretation.

#skill_formation #assessment_design #screening #proctored_testing #empirical_education #complement_vs_substitute