ScienceNot yet confirmed elsewhere1 publisher2 min readPublished
Explaining itself made the AI more persuasive and the reviewers less right
In a Harvard, MIT and University of Washington experiment, 228 reviewers deferred to an LLM at the same rate with or without a rationale. Only the silent version improved their accuracy.
The Scientist · Science desk
What happened
- Researchers tied to Harvard Business School, MIT and the University of Washington had 228 experienced evaluators score nearly 50 submissions to an MIT challenge.
- Three conditions were tested: no AI help, an LLM verdict with a written rationale, and a black-box pass or fail with no reason given.
- Reviewers sided with the LLM about 75 percent of the time in both AI conditions, against 54 percent for human-sourced decisions.
- The black-box verdicts moved decisions closer to the four-expert baseline; the explained verdicts did not.
- When the model explained a rejection, agreement spiked: fewer false positives, and substantially more false negatives.
Why it matters
- contradiction Explainability is procured as the feature that keeps a human engaged, yet the opaque condition was the one that produced better decisions, inverting the buying logic.
- constraint A saved model rationale cannot double as proof of human review, because the rationale is what displaced the review.
- exposure Funnels instrumented for bad bets approved will not register good bets quietly declined, so the induced error accumulates off the dashboard.
- decision Anyone running an AI-assisted screening pipeline now has to choose whether to log verified overrides, since without that field deference and judgment look identical in the record.
The rationale bought no additional compliance. Deference ran at about the same level whether the model argued its case or simply stamped a verdict, so what the written reasoning changed was the composition of the errors, not the amount of following [17]. That is the result with operational consequences, and the researchers say it contradicts the common assumption that model explanations augment human decision-making [14].
The 21-point gap between how often these reviewers went along with machine advice and how often they went along with human advice is the figure worth carrying into any screening design [16]. Direction matters more than volume, though. A rejection commits no budget and creates no owner, and the researchers reach for Xerox shutting down early Ethernet and PostScript work as their illustration of that cost, against Google Glass and Amazon's Fire Phone on the false-positive side [9]. Only one of those two failure modes leaves a project to write a post-mortem about.
The mechanism the authors offer is about accountability, not model quality. People weigh negative information more heavily than positive, a pattern the paper labels negativity bias [10], and as they put it, "Rejection is an active, eliminative decision that feels more consequential and accountable than preserving optionality," while also maintaining the status quo and requiring no resource commitment [11]. A fluent paragraph explaining why a proposal falls short hands the reviewer a ready-made justification for the decision they were already disposed toward, and the study reports evaluators leaning on surface cues such as fluency and coherence [12]. The authors describe narrative explanations as suppressing productive overrides, meaning the cases where someone independently verifies a persuasive model output before agreeing with it [15][4].
That distinction is where enterprise exposure sits. Screening under time pressure with only basic information is precisely the job LLM recommenders are being handed [18]. A stored rationale for each decision is an attractive artifact because it looks like a control and files like evidence. What it evidences is that a reason was available, not that anyone tested it.
The study's ground truth deserves stating plainly: correct means agreeing with four human experts on one challenge's submissions, so this measures alignment with a small panel rather than commercial outcomes [3]. Within that frame, the authors end on design rather than warning, writing that "effective human-AI collaboration requires designs that preserve rather than supplant independent human judgment" [13]. On this evidence, the explanation layer is a persuasion feature that has been filed under governance.
What to watch
- Whether the effect replicates outside early-stage innovation screening, and whether the black-box advantage survives peer review.
- Whether the same reversal shows up on approve-side recommendations, where negativity bias should not be doing the work.
- Whether a follow-up varying explanation length or confidence hedging moves the false-negative rate.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence42
- Adoption
- Insufficient
- Hype gap+18
- Incentives34
- Confidence45
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Researchers associated with Harvard Business School, MIT and the University of Washington ran an experiment in which 228 experienced evaluators assessed nearly 50 submissions to an MIT challenge.
- [2]
The experiment tested three scenarios: human-only proposals with no AI assistance; LLM evaluations with a written rationale for the decision; and black-box AI pass/fail recommendations with no accompanying explanation.
- [3]
Evaluators' decisions were compared to decisions made by four human experts, which were treated as the correct baseline.
- [4]
Responses were classified as outright compliance with the LLM recommendation, overrides, or productive overrides, meaning the evaluator independently verified persuasive model outputs before deciding.
- [5]
Overall, the evaluators accepted LLM recommendations 67% of the time.
- [6]
Evaluators agreed with both black-box and narrative LLM decisions roughly 75% of the time, but agreed with human decisions only 54% of the time.
- [7]
When given an LLM recommendation to reject a submission plus a reason why, evaluators disproportionately agreed, which reduced false positives but substantially increased false negatives.
- [8]
Black-box recommendations improved the quality of decisions, aligning them with human experts, while recommendations accompanied by narratives did not.
- [9]
The researchers cite Google Glass and Amazon's Fire Phone as false-positive examples, and Xerox terminating early Ethernet and PostScript projects as a false-negative example.
- [10]
The researchers noted that people are cognitively predisposed to weigh negative information more heavily than positive information, a phenomenon known as negativity bias.
- [11]
The researchers wrote: "Rejection is an active, eliminative decision that feels more consequential and accountable than preserving optionality." They add that it maintains the status quo, avoids risk and bias, and requires no resource commitment.
- [12]
LLM explanations provide ready-made justifications for going along with rejection decisions without independent verification, and evaluators often rely on surface cues such as fluency and coherence, according to the researchers.
- [13]
The researchers wrote: "Our findings reveal that LLM explanations do not necessarily improve decision-making" and "Effective human-AI collaboration requires designs that preserve rather than supplant independent human judgment."
- [14]
The result contradicts a common assumption that LLM explanations augment human decision-making.
- [15]
The researchers posit that narrative explanations suppress productive overrides because the LLM provides a convincing argument that is easy to accept, discouraging independent human verification.
- [16]
The gap between agreement with LLM recommendations and agreement with human decisions was 21 percentage points.
- [17]
Because agreement rates were roughly equal in both LLM conditions while alignment with the expert baseline diverged, the written rationale changed which errors evaluators made rather than how often they deferred.
- [18]
Because decision-makers have limited time and only basic information, they are increasingly turning to LLMs that generate recommendations and rationales from context.
Sources
1 independent publisher whose own reporting we read for this story.
- cio.comWhen AI explains its decision, humans may stop thinking independently | CIO
1 article · August 22, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- AI Explainability and Explanation DesignFollow
- Human-AI Decision CollaborationFollow
- Automation BiasFollow
- Enterprise AI GovernanceFollow
- Early-Stage Innovation ScreeningFollow
Entities
- Harvard Business SchoolFollow
- MITFollow
- University of WashingtonFollow
- CIO.comFollow
- Google GlassFollow
- Amazon Fire PhoneFollow
- XeroxFollow