Repeating an AI classroom trial with 164 students erased the training advantage

The 2024 cohort was small. Sixty-six students split three ways is about 22 per arm ^[1], or roughly five subgroups per condition at teams of four or five ^[2]. The trained group's clearest lead was on the take-home exam ^[14], measured on that population. The 2025 repeat enrolled 164 students ^[7], about 55 per arm ^[3], and the three groups came out level ^[15]. Neither year's write-up gives scores or effect sizes ^[4].

The two instruments disagree about the same students. In the timed exercise, every subgroup in the unguided arm kept at least one misleading or legally extraneous term from ChatGPT's output ^[12], and some swapped "shall" and "individual" for AI-suggested alternatives without showing they understood the legal implications ^[11]. Schrepel expected that to carry into the exams, and it did not; the errors did not repeat, and the unguided group scored slightly higher than the AI-free group ^[19]. "I was wrong," he writes ^[20].

His explanation is that students learn to spot AI weaknesses through their own use, especially when accuracy has real consequences ^[21]. That is one instructor's reading of his own cohort. The two settings work differently: the exercise put ChatGPT's revisions directly inside the text under a 20-minute clock ^[2][4], and accepting a suggestion that is already in the document takes no keystrokes. The exams were a multiple-choice test and an individually graded take-home revision ^[6]. The arm with no tool at all did get something out of the constraint, which Schrepel records as deeper discussion among group members ^[9].

For the year-one result to transfer, your population has to look like the year-one population. Schrepel attributes the 2025 convergence to growing chatbot familiarity, and says many students already use these tools in daily life ^[16]. A team with thin baseline use is in the 2024 condition; a team where everyone already prompts daily is in the 2025 condition. The second transfer condition is the grader. Substance, clarity, proportionality and innovation ^[2] are judged by a person, and no test suite fails when a drafter keeps a legally extraneous term. Code review has a ground truth this task lacks, so the self-correction Schrepel observed may have a different cause.

Whether the advantage comes from structured training or just hands-on experience remains an open question, Schrepel says ^[18]. His policy conclusions do not wait for the answer: department leaders should resist blanket bans and give instructors room to experiment, and universities should invest in AI skills for faculty, many of whom feel poorly equipped ^[22]. He adds that ethical use and legal responsibility still have to be taught ^[25]. He also argues that pure literature reviews should no longer count toward a master's degree, and that programmes should require practice-oriented or empirical work in which thoughtful AI use is part of the grade ^[23].