AI built stronger judgment only in senior patent attorneys over a 90-day trial

"Judgment is what sets senior professionals apart from their junior peers," David Autor and Tanya Rodchenko wrote ^[10]. In their account, it takes thousands of hours on the job to build, usually through tedious, routine work done under supervision ^[11]. The routine work in this trial was patent drafting. Each lawyer drafted a patent from simulated inventor materials after 10 days and again after 90 ^[8].

The drafting pattern was clearest for juniors, who also saved time ^[16]. They finished the 10-day draft in about 14.5% less time ^[19], 18 minutes under the control group's 124-minute average ^[16]. A grader reading an assisted draft sees a better document. The draft alone cannot show whether the junior could have produced it without help.

Patent lawyers also review each other's work, so the authors added a redline task at 90 days. Subjects had to mark up and correct a hypothetical patent seeded with substantive and stylistic errors ^[6]. The critical mistakes, sometimes called "patent profanity," include overclaiming an invention's novelty or scope, and they can make a patent unenforceable in court ^[7].

On that task, senior lawyers who had used the tool for 90 days showed stronger judgment than seniors who had not ^[9]. Junior judgment did not improve on average, and more juniors landed at each end of the scale ^[2]. The authors' headline finding does not put a size on either tail.

These are results from one workload. The tool was a then-unreleased Google Labs patent assistant, now part of Gemini Notebook, and all eleven firms do regular, non-exclusive business with Google ^[14]. Two-thirds of 133 is about 89 lawyers with the tool and 44 without ^[20]. The junior and senior findings come from subsets of those groups. For the numbers to carry to another team, its juniors' routine work would have to resemble patent drafting, and its review test would have to look like the redline.

The authors argue that the question needs months of sustained use in ordinary work, a credible test of unassisted judgment, and blinded grading by domain experts, and that the three are rarely found together ^[18]. Three months of routine use ^[1] is the expensive part. I think it is the main reason this result deserves more weight than a short lab session would get.

The authors place the finding against two sets of earlier studies. In radiology, business problem-solving, job-seeker writing and legal education, AI built into training improved less-experienced workers' independent performance ^[12]. With software engineers, management consultants, high school students and clinicians, gains often faded once the tool was removed. There, the authors wrote, AI "serves as a temporary exoskeleton that boosts immediate output" ^[13]. The juniors in this trial show both outcomes inside one cohort, using one tool.

This trial randomized access to the tool ^[14]. Positive training results come from the other studies, where the tools were "thoughtfully embedded into training workflows" ^[12]. So the case for redesigning how juniors use AI rests on those studies, plus this trial's evidence that access alone left average junior judgment flat ^[2]. In my view, a firm that trains juniors through routine drafting needs an unassisted review test, something like the redline, to find out which end of the scale each junior is on.