ScienceNot yet confirmed elsewhere1 publisher2 min readPublished
Researchers sharing unpublished work with chatbots have only account settings to protect it, expert says, as OpenAI and Anthropic deny training on two uploads
OpenAI and Anthropic each deny training on data an NYU mathematician and a Copenhagen biologist uploaded, Scientific American reports. A Stanford privacy researcher says the companies' practices are too opaque for experts to judge either case.
The Scientist · Science desk
What happened
- OpenAI had claimed an internal model solved one of math's seven million-dollar problems when an NYU mathematician insinuated the company had stolen progress he shared with one of its chatbots.
- Weeks later the New York Times reported a similar story involving Anthropic and a University of Copenhagen biologist, this time over a group of enzymes.
- Researchers keep using the models as a soundboard, for literature review and to polish nearly finished work, and mathematicians and theorists use them to generate entire proofs.
- Scientific American says there is no evidence that a rogue agent took users' conversations in the case of any of the companies' scientific discoveries.
Why it matters
- exposure Personal-account users carry the exposure. King says enterprise accounts seem safe, but on a personal account "all bets are off", and in many cases users must opt out explicitly.
- cost Abstaining has a price too. The magazine describes a choice between using the models and risking a leak, or not using them and risking that someone who does publishes first.
- precedent Both companies are stepping into math and science ahead of going public, so we'd expect each new research result they announce to meet the same provenance question the NYU and Copenhagen researchers raised.
A denial about training covers one use of an upload [3]. Scientific American describes a second route that such a denial would not cover. Some researchers suspect the companies may not have the full picture themselves, and that their agents could be reaching data in ways the companies do not realize [17].
Jennifer King, who studies data privacy and policy as a fellow at Stanford's Institute for Human-Centered Artificial Intelligence [14], put the underlying problem this way: "Right now, these systems infer things about you without telling you that they're inferring things about you," she said [5].
The thing this doesn't tell you is how often a suspicion like this turns out to be right. The NYU and Copenhagen episodes are two cases, each a researcher's suspicion set against a company's denial [1][2][3]. Nothing in them lets an outsider compare what a model would have produced with and without the uploaded work, so neither works as a control.
Clement Canonne, a theoretical computer scientist at the University of Sydney [13], has opted out. "I'm not encouraging it, and I'm not doing it myself," he said. "I'm not using them." [9] The magazine describes a wider drift, with some researchers now seeing the models more as competitors than tools [12].
In our view the record supports a narrow conclusion: a user has no reliable way to confirm how an upload is handled [15]. A broader one, that either company used the mathematician's or the biologist's work, would need evidence the magazine does not report.
What to watch
- Whether either company releases records or an independent audit showing how the NYU mathematician's and Copenhagen biologist's uploads were handled.
- Any change to default training settings on personal accounts, where many users now have to opt out explicitly.
- Whether more researchers go public with provenance complaints as OpenAI and Anthropic announce further research results.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence30
- Adoption
- Insufficient
- Hype gap+40
- Incentives45
- Confidence40
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
When OpenAI claimed an internal model had solved one of math's seven million-dollar problems, a mathematician at New York University insinuated that the company had stolen his progress, which he had shared with one of their chatbots.
- [2]
Weeks later, the New York Times reported a similar story between Anthropic and a biologist at the University of Copenhagen regarding a group of enzymes.
- [3]
Both OpenAI and Anthropic stated unequivocally that they did not train their models on the mathematician's and biologist's uploaded data, respectively.
- [4]
There is no evidence that an agent going rogue and taking individual users' conversations with chatbots happened in the case of any of the AI companies' scientific discoveries.
- [5]
"Right now, these systems infer things about you without telling you that they're inferring things about you," King said.
- [6]
King says the only way for researchers to ensure their safety when using large language models at present is to look at their account settings carefully; enterprise accounts seem to be safe, but "if you are on a personal account, all bets are off," she warns.
- [7]
Users on personal accounts need to explicitly opt out in many cases to prevent their information being used for training.
- [8]
"The companies are not transparent about these practices, so it's really hard to pin down," King says; that means experts can't speculate about any particular case.
- [9]
"I'm not encouraging it, and I'm not doing it myself," said Canonne. "I'm not using them."
- [10]
Scientists use the companies' models as a soundboard, as a tool for literature review, and for cleaning up nearly finished work, and mathematicians and theorists use them to generate entire proofs for their research.
- [11]
Scientists may feel they face a choice: either use the models and risk leaking a discovery, or don't and risk someone who does publishing it first.
- [12]
Some researchers are now seeing the models more as competitors than tools.
- [13]
Clement Canonne is a theoretical computer scientist at the University of Sydney.
- [14]
Jennifer King is a fellow at the Stanford University Institute for Human-Centered Artificial Intelligence who researches data privacy and policy.
- [15]
Editorial inference: because King says personal-account users face 'all bets are off' and that the companies are not transparent about these practices, a user has no reliable way to confirm how an upload is handled.
- [16]
Ahead of taking their companies public, OpenAI and Anthropic are increasingly stepping into math and science to show how capable their products are.
- [17]
Some suspect the companies may not have the full picture themselves, and that their agents could be accessing data in ways the companies don't realize.
Sources
1 independent publisher whose own reporting we read for this story.
- scientificamerican.comScientists worry AI is stealing their ideas
1 article · October 13, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- AI in scientific researchFollow
- AI Data PrivacyFollow
- Training-data consent as discountFollow