Penn's 140,000-article news dataset shows a publisher's lean varying by topic
Duncan Watts's Penn team released more than 140,000 articles from the 2024 U.S. election year, annotated by AI, with a Science Advances paper. The study found that a publisher's political lean can vary considerably by topic, a detail that gets averaged away in a single bias score.
The Scientist · Science desk
What happened
- The Media Bias Detector launched in 2024 from Penn's Computational Social Science Lab, a joint venture of Penn Engineering, the Annenberg School and Wharton.
- The tool uses AI to analyze articles for topic, tone, political lean, and the facts and people included in coverage.
- Its framework splits coverage into measurable components, so researchers can study how editorial choices shape what readers see and what they don't.
- Lead researcher Samar Haider said the paper evaluates the annotation pipeline and shows findings the public dashboard cannot yet display.
Why it matters
- What's now possible capability Outside researchers can now ask their own questions about selection and framing at article level, using data that Watts says is richer than the dashboard can show.
- What it limits constraint Scores built from signatures of bias describe how outlets differ from each other or from their own past. They cannot certify any outlet as accurate.
- Who has to decide decision Anyone ranking outlets by one lean number now has to justify averaging over topics, since the study found lean varies considerably by topic.
Measuring bias needs a reference point, and for most news there isn't one. "The fundamental challenge with measuring bias is that, by definition, it requires one to know the 'true' value of the thing you're measuring," Duncan Watts said. "And, for most topics in the news, objective ground truth simply isn't available." [6]
The team's workaround is to look for what Watts calls signatures of bias: systematic differences in the way publishers select and frame information [7]. Watts said: "By aggregating these signals, and comparing them across publishers, or even within publishers over time, our hope is that we can detect the signatures of bias without having to decide what a perfectly unbiased publisher would look like." [8]
The most concrete result in the write-up is that a publisher's political lean can vary considerably depending on the topic, a variation that a single bias score averages away [9]. The phys.org account does not give the size of that variation, which outlets show it, how the articles were chosen, or the results of the evaluation. Because an AI model does the annotating [2], that evaluation of the annotation pipeline [5] is the check on the measuring instrument.
The dataset covers articles from the 2024 U.S. presidential election year [3], so it cannot show whether the topic-level pattern holds in other years. Coverage in a year without a presidential race may split differently across topics.
Watts framed the release as an opening. "This paper provides just a taste of what is possible with this new class of large-scale but still very granular news data," he said [10].
The paper is Samar Haider et al., "The Media Bias Detector: A framework for annotating and analyzing the news," Science Advances (2026), DOI 10.1126/sciadv.aea7456 [12].
What to watch
- The paper's evaluation results for the annotation pipeline, which would show how reliable the AI labels are.
- Whether the team moves the findings Haider says the dashboard cannot yet show onto the public dashboard.
- Whether other groups use the 2024 dataset to test the topic-level lean result, and whether it holds in a non-election year.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence50
- Adoption
- Insufficient
- Hype gap−10
- Incentives
- Insufficient
- Confidence45
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
The Media Bias Detector was introduced in 2024 as a collaborative Penn venture conceived in Penn's Computational Social Science Lab, a joint venture of Penn Engineering, the Annenberg School for Communication and the Wharton School.
- [2]
The Media Bias Detector uses artificial intelligence to analyze news articles for characteristics including topic, tone, political lean, and the facts and people included in coverage.
- [3]
In a new Science Advances paper led by Duncan Watts, the team introduces the scientific framework behind the Media Bias Detector and releases a data set of more than 140,000 articles published during the 2024 U.S. presidential election year.
- [4]
Rather than asking whether an outlet is simply biased, the framework breaks news coverage into measurable components, allowing researchers to study how editorial choices shape what readers see and what they don't.
- [5]
Lead researcher Samar Haider, a Ph.D. student in computer and information science, said the paper presents a rigorous evaluation of the data annotation pipeline and illustrative findings from the underlying data that cannot yet be visualized on the dashboard.
- [6]
"The fundamental challenge with measuring bias is that, by definition, it requires one to know the 'true' value of the thing you're measuring," says Watts. "And, for most topics in the news, objective ground truth simply isn't available."
- [7]
The researchers look for what Watts calls the signatures of bias: systematic differences in the way publishers select and frame information.
- [8]
"By aggregating these signals, and comparing them across publishers, or even within publishers over time, our hope is that we can detect the signatures of bias without having to decide what a perfectly unbiased publisher would look like," says Watts.
- [9]
The study found that a publisher's political lean can vary considerably depending on the topic, a level of nuance not captured by a single bias score.
- [10]
"This paper provides just a taste of what is possible with this new class of large-scale but still very granular news data," says Watts.
- [11]
Watts said the data set underlying the dashboard is much richer than any dashboard can adequately capture.
- [12]
The paper is Samar Haider et al., The Media Bias Detector: A framework for annotating and analyzing the news, Science Advances (2026), DOI 10.1126/sciadv.aea7456.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- phys.orgAI helps detect media bias, informing voters and news consumers
1 article · October 15, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.