Two design choices travelled together in the condition that worked. When a student got a problem wrong, NUMI walked them through the mistake instead of supplying the solution [3], and then required them to demonstrate the same skill correctly three times in a row [4]. That pair was measured against conventional computerized instruction [4], which means the three points cannot be apportioned between the withholding and the mastery gate [17]. Both are pedagogy, and only one of them is about answers.
The framing that reaches most readers is conditional: an AI tutor helps only if it holds back answers and coaches the student through the problem [16]. Oreopoulos grounds that in behaviour rather than technology, since students want to finish assignments quickly and, as he puts it, "easier equals less effort and less learning" [10]. But the reported experiment contains no condition in which an AI tutor does hand over the solution [18], so the conditional is a principle the results are consistent with rather than a contrast the study measured. Against the accumulating reports that AI harms learning by producing answers on demand [15], it is a reasonable principle to hold, and it remains a principle.
Three points is the number, and the account does not say three points out of how many, or how that compares with the spread of scores [5][6]. With more than 6,000 students in the sample [1], a difference that small can be statistically clean while sitting near the edge of what a teacher would notice, which is close to how Oreopoulos describes it when he refuses the game-changer language [8]. The write-up also does not say whether assignment happened at the student, classroom or school level [20], and in a school-delivered intervention that detail governs how much of the apparent precision survives clustering. Oreopoulos calls the work a proof of concept and possibly the first objective evidence that a well-designed AI tutor beats instruction without AI [7]; the paper sits alongside two other recent working papers from the same line of research at the National Bureau of Economic Research [11].
His choice of comparison is the part worth carrying forward. He does not claim NUMI beats a good human tutor. He says a good human tutor is usually better, and that the question worth asking is what a well-designed AI tutor does for a student who is stuck with nobody available to help [9]. That is a defensible baseline, and it is the one in which a modest gain counts for something. It is also the baseline that makes the mastery loop, not the model, the object anyone should be trying to reproduce.
