Wrong AI advice shown before a person decides lowers that person's accuracy
Participants judging defendants in two experiments grew less accurate when a supposed AI gave wrong advice before they decided. A 2022 study in which an accurate parser cost annotators no quality suggests the risk grows with the model's error rate.
Written for builders.
What happened
- Across two experiments, the researchers varied when participants got support from a supposed AI system, either before or after they made their own judgments.
- Experiment 2 was preregistered, and the data and materials for both experiments are freely available on the Open Science Framework.
- In a 2022 LREC study, annotators given a high-accuracy parser's pre-annotation worked faster and more consistently than fully manual annotators, with no drop in quality.
- Political institutions recommend keeping a human in automated decision processes to guard against erroneous or biased algorithmic decisions.
Why it matters
- What it limits constraint A reviewer who sees the model's answer first is weakest on the cases that most need review, the ones where the model is wrong, so the human safeguard protects less than policy assumes.
- Who has to decide decision Teams building review queues have to choose an order: record the reviewer's own call before showing model output, and give up some of the speed and consistency that pre-annotation brings.
- Who's exposed exposure Defendants in the countries using judicial decision support depend on a human check that, in this experiment, gave way under bad advice delivered first.
Because the experiments moved only the timing of the support [2], the larger accuracy loss when incorrect advice came first [3] points to order itself. We think this is anchoring in the ordinary sense: the first answer a person sees shapes the one they give. The authors place the result alongside more than two decades of research describing a human tendency to use support-system output as a shortcut to avoid searching for or processing other information [9].
The cases were simulated [1]. Real judicial systems usually gather and summarise information or recommend a decision for a human to act on [11].
At first glance the 2022 LREC paper by Mikulova and colleagues [8] looks like a counterexample, since pre-annotation in that study cost no quality [7]. We think the two results fit together. The judicial experiments tested advice that was wrong [3]. The parsing study took its suggestions from a high-accuracy parser [6]. Higher consistency is also what anchoring would produce if annotators accept the same parser tree.
In our view the angle holds, with one condition: a reviewer who is shown the model's answer first is a check whose value falls as the model's error rate rises [3][7]. Neither abstract reports an effect size, so these two papers do not show the error rate at which suggestion-first review stops paying for itself.
What to watch
- Effect sizes and the share of incorrect-advice trials in the posted Open Science Framework data for the defendant-judging experiments.
- A study that varies model accuracy within one annotation or review task, which would locate the error rate at which pre-annotation starts to cost quality.
- Whether human-in-the-loop guidance starts requiring reviewers to record their own judgment before they see the system's recommendation.
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
In two experiments, the researchers simulated an automated decision-making process in which participants judged multiple defendants in relation to various crimes.
- [2]
The researchers manipulated the time at which participants received support from a supposed automated system with Artificial Intelligence: before or after they made their judgments.
- [3]
Human judgment was affected when participants received incorrect algorithmic support, particularly when they received it before providing their own judgment, resulting in reduced accuracy.
- [4]
The data and materials for the experiments are freely available at the Open Science Framework; Experiment 2 was preregistered.
- [5]
Political institutions recommend the presence of humans in automated decision-making processes as a safeguard against potentially erroneous or biased algorithmic decisions.
- [6]
The study compared dependency syntax annotation by annotators using a pre-annotated version produced by a high-accuracy parser with fully manual annotation.
- [7]
The experiment confirmed that pre-annotation is an efficient tool for faster manual syntactic annotation which increases the consistency of the resulting annotation without reducing its quality.
- [8]
Quality and Efficiency of Manual Annotation: Pre-annotation Bias, by Marie Mikulova, Milan Straka, Jan Stepanek, Barbora Stepankova and Jan Hajic, appeared in the Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC), Marseille, June 2022.
- [9]
For more than two decades, the scientific literature has pointed to a human tendency to use the information provided by support systems as a shortcut to avoid searching for or processing other information.
- [10]
Countries using automated decision support systems in the judicial context include the United States, the United Kingdom, China, Estonia, Argentina, Poland and Spain.
- [11]
In these cases the system usually does not decide fully autonomously but supports the human decision, for example by gathering and summarizing information or recommending a particular decision.
Sources & coverage · 2 publishers
The reporting this story was synthesized from, earliest first. Every link goes to the original.
- aclanthology.orgJun 10Quality and Efficiency of Manual Annotation: Pre-annotation Bias - ACL Anthology
- pmc.ncbi.nlm.nih.govJan 6The impact of AI errors in a human-in-the-loop process - PMC