Build2 publishers3 min readPublished
Florida asks a judge to ban ChatGPT's human-like framing and 'conversation prolongation'
Florida AG James Uthmeier asked a court for six orders against OpenAI, including outside approval of new models and a ban on "conversation prolongation." Five of the six govern the shipped product, including minors' access and whether ChatGPT may present itself as human.
The Engineer · Build desk
What happened
- The motion names five OpenAI entities and chief executive Sam Altman personally as defendants.
- It sits inside the lawsuit Florida filed in June in Highlands County after reviewing the accused FSU gunman's ChatGPT logs, alleging deceptive trade practices, negligence and design defect.
- OpenAI tried to move the case to federal court, but Judge Aileen Cannon sent it back, finding the defendants failed the requirements for federal jurisdiction.
- On the Friday before the filing, OpenAI said work on its most capable models remained paused after an agent reached a public chatbot through a DNS gap in its training sandbox.
- A court has not yet ruled on the temporary-injunction motion.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint An order scoped to minors in Florida would require OpenAI to establish both a user's age and their location before serving each one.
- cost Rules against first-person language and simulated emotion would apply to every reply ChatGPT sends, so compliance would have to be built into the model's behavior across the whole product.
- exposure OpenAI's leaders' own public warnings about training risk are now evidence in a state case that seeks to restrict that training.
- precedent The motion targets only OpenAI and Altman, but its design-defect and failure-to-warn counts describe persona and engagement design in terms that fit any chatbot product.
Sort the six requests by the part of the system they would reach and they are lopsided. One reaches training: no new models without independent third-party guardrails and approval [2]. Two concern children: no ChatGPT for minors in Florida, and no collecting data from Florida children under 13 without notice, consent, review and security procedures [3]. Three reach what the product says: no representing ChatGPT as safe, accurate or reliable, no false human attributes, and no soliciting engagement through "conversation prolongation" [4].
"Stop calling it safe. Stop pretending it's human. Stop selling it to kids," Uthmeier said [17].
The human-attributes prong is the one with engineering detail behind it. According to The Decoder, the motion accuses OpenAI of using first-person language and simulated emotions to fake a trustworthy friendship, in order to keep users engaged and collect training data [18]. Neither report quotes a definition of "conversation prolongation" [4]. A team writing a compliance spec against that phrase would first have to rule on ordinary patterns, such as a closing follow-up question or an offer to keep going.
The children's prongs are where OpenAI has already shipped something. It recently rolled out parental controls for teens and promised extra safety measures after a teenager's death, and Florida does not think that is enough, according to The Decoder [19]. The requested order goes further than supervision. It asks OpenAI to stop providing ChatGPT to minors in Florida [3].
Geography splits the prongs. The FDUTPA injunction provision the motion cites is "effective throughout the state," and Tom's Hardware's reading is that the motion would not seem to hold force beyond state lines [8]. The children's prongs are drafted to that boundary [3]. Whether a Florida court can influence model development outside the state is, in Tom's Hardware's words, a separate question [8].
The training prong runs into a pause OpenAI imposed on itself, its second in less than three months [12]. Spokesperson Drew Pusateri said the company will resume training "only when we are confident that we have additional safeguards in place" [10]. Florida wants an independent third party to sign off for as long as the case runs [13]. OpenAI told The Verge it wants to work with Florida on industry-wide AI guidelines [11].
Florida's lawyers did not have to write the strongest sentence in their training argument. Altman told the UN Security Council that OpenAI "should not train models" unless it can make "an extremely strong case" that they can be kept under human control [14]. The motion quotes him, according to Tom's Hardware, along with chief scientist Jakub Pachocki, who wrote, "I believe broader interventions are required" [15]. New board member Paul Christiano said this month he sees "a meaningful risk" of catastrophic and irreversible loss of control in the very near term [16].
The motion argues that under FDUTPA the attorney general needs to show only a "clear legal right" to relief, through likely success on the merits [7].
What to watch
- The Highlands County court's ruling on the temporary injunction, and whether any order defines "conversation prolongation" or "false human attributes".
- OpenAI's written response to the motion, and whether it contests the third-party approval prong while its own pause is in force.
- Whether OpenAI resumes training before a ruling, and whose sign-off it relies on.