Invest1 publisher2 min readPublished
ChatGPT users sue to make OpenAI get consent before contractors read their chats
Two ChatGPT users are suing OpenAI to make human review of chats opt-in, a change that would reach more than 900 million weekly users. Until a court rules, companies should assume staff chats on consumer ChatGPT, with the default setting on, can reach a staffing-firm contractor.
The Investor · Invest desk
What happened
- Two California ChatGPT users filed a proposed class action against OpenAI in the Northern District of California, alleging contractors read real chats without proper disclosure.
- Under OpenAI's Project Lily, contractors hired through staffing firms read real conversations, summarize the user's goal and score four model answers from 1 to 7.
- 404 Media, which first reported the program on September 14, found reviewers see a 'user memories summary' that can reveal location, profession or personal life despite stripped usernames.
- The plaintiffs want opt-in consent before any outside reviewer sees a chat and the 'Improve the model for everyone' setting switched off by default.
- OpenAI has until October 13 to file its response to the complaint in court.
Compiled by The InvestorSomething wrong?How this is made
Why it matters
- constraint An opt-in rule would limit OpenAI's human graders to chats from users who agree, so its anti-sycophancy training would draw on a smaller, self-selected pool of conversations.
- precedent If a court finds staffing-firm graders are outside parties a privacy policy must name, any chatbot maker using outside human reviewers for RLHF faces the same disclosure test.
- cost The retraining request is the one with an open-ended bill, since its cost scales with how many OpenAI models were trained on Project Lily grades.
A single Project Lily review yields more than one grade. Four scored answers can be sorted into as many as six head-to-head pairs [1], and those preferences get fed back into training so the model learns which responses people prefer [5]. OpenAI has said the reviews target two behaviours, the chatbot acting too human-like and the chatbot being too agreeable [8]. A grader cannot judge whether an answer flatters someone without reading what that person wrote, and under Project Lily the contractors read full conversations as well as the prompts [3].
Those contractors come through third-party staffing firms [3]. That makes each reader an outside party. The plaintiffs argue OpenAI's privacy policy names some categories of outside recipients without ever flagging data annotation or human evaluation vendors [10]. On that omission they build eight claims, among them California's Unfair Competition Law, its Consumer Privacy Act and the common-law tort of intrusion upon seclusion [11], and they seek damages, restitution and punitive damages [12]. Decrypt's account of the complaint does not include a damages figure.
For a company whose staff use consumer ChatGPT, the filter allegation matters most. OpenAI runs conversations through an automated system before any human sees them, but the complaint alleges the filter does not always catch everything and personal details sometimes reach contractors [6]. The product has more than 900 million weekly users [9]. Decrypt reports that people type in health symptoms, tax questions and legal troubles [16].
OpenAI's response is due October 13 [15]. It can defend its current disclosure, settle on new disclosure, or fight the default change the plaintiffs want [13]. I think a disclosure settlement is the likeliest early outcome, or rather the narrow version of one: the in-chat warning the plaintiffs already request [14], plus a line in the privacy policy naming human evaluation vendors. That package answers the omission the complaint is built on and keeps the grading pipeline running. The punitive damages request [12] cuts the other way. It gives the plaintiffs room to hold out for the default change. If OpenAI switches "Improve the model for everyone" off by default before any ruling, I have this wrong.
What to watch
- OpenAI's October 13 filing, and whether it offers new privacy-policy wording naming human evaluation vendors or defends the current text.
- Whether the deletion-and-retraining request survives OpenAI's response, since it is the only remedy that reaches models already trained.