Skip to content

Leadership2 publishers3 min readPublished

California prices negligent child harm at up to $1m per child

Gavin Newsom signed a package on Thursday that bans autoplay and history-based feeds for under-16s and requires companion chatbots to alert parents when a young user shows distress. Penalties are counted per child.

The Board Room · Leadership desk

Photograph accompanying California prices negligent child harm at up to $1m per child
Photo: theguardian.com

What happened

  • The package bans autoplay scrolling and algorithmic feeds based on user history for people under 16, with AB 1709 carrying the infinite scroll and autoplay restriction.
  • Companies running AI chatbots must enact crisis protocols that notify parents when young users show signs of emotional distress.
  • The new regulations extend California's child sexual exploitation laws to cover digitally altered and AI-generated sexual imagery of minors.
  • Newsom signed separate laws on Wednesday, supported by Anthropic, requiring state rules for independent evaluators of AI model safety risks and a registry of financially independent AI auditors.

Compiled by The Board RoomSomething wrong?How this is made

Why it matters

  • cost Counting by the child turns a ranking default into a variable legal cost that scales with cohort size: at the statutory ceiling, one finding covering a thousand California minors is a billion dollars against reserves.
  • constraint Both the under-16 feed rules and the chatbot notification duty depend on knowing which account belongs to a minor, and the published accounts specify no method, so cohort identification is the schedule risk, and its timeline is still open.
  • contradiction Meta says personalization is how it delivers age-appropriate content to teens; the EFF says stripping those tools cuts minors off from speech. Two opposed complaints describe one product change.
  • decision Chatbot teams now have to set a distress threshold and decide what a parent is told, because for the first time the duty runs to someone outside the account.

CBS reports the package penalizes large social media companies up to $1 million per child where they are found negligent of harming children through their platforms [2]. At that ceiling, a finding covering a thousand minors comes to $1 billion [21]. For scale, the up-to-$18 billion child-safety settlement CBS reports from last month, reached with 28 other states after a 2023 lawsuit alleging a tech company deliberately designed features to addict children, is 18,000 children at California's per-child cap [14][22]. Both figures are ceilings, and the California one attaches only where negligence is found [2].

The timing is the open question. Neither account of the signing gives an effective date for the feed rules or the chatbot duty [23]. The published accounts also skip how a platform must establish which users are under 16 [24]. A team can begin the cohort work this quarter without a compliance date to build toward. That determination is the load-bearing engineering: AB 1709's ban on infinite scroll and autoplay applies to users under 16 [7], and the chatbot crisis protocol is triggered by young users showing signs of emotional distress [4].

"This is about actually addressing the problem, the scrolling, the algorithms. This is about addressing the substantive issue around the engineering," Newsom said [6]. The objections arrive from opposite directions and describe the same product change. Meta raised concerns about some of the bills, saying the restriction on addictive feeds hinders its ability to bring enjoyable content to users on its platforms [12]. "Personalization is also how we deliver age-appropriate content for teens that is relevant to them," spokesperson Jim Cullinan said in a statement [13]. The Electronic Frontier Foundation reads AB 1709 as a "functional ban on social media use" for users under 16 [10]. "California should be passing laws to ensure that technology really works for people of all ages, not enacting social media bans that cut young people off from digital lifelines, communities, and speech," said Rindala Alajaji, the group's associate director of state affairs [11]. Either way, the mechanics are the same: for under-16 accounts, ranking on a user's own history goes away [3].

The record does not report a filed challenge to that design mandate, and the EFF's language is a preview of an argument [10]. The chatbot obligation also sits on different ground. It requires an operator to notify a parent when a young user shows signs of distress [4]. The work there is escalation design: detection thresholds, who gets paged, and what the parent is told.

One of the chatbot measures was inspired by the death of Adam Raine, whose mother said he died because he was "coached" by a chatbot [17]. "Adam was an early adopter of AI, and so many of us parents did not understand the dangers back then," Maria Raine said at a news conference [18]. The governor's office says California now has the "strongest companion chatbot regulations" in the United States [8].

The direction of travel is legible in what Newsom has already signed and refused. Last year he signed a law requiring platforms to remind users they are interacting with a chatbot and not a human [15]. Hours later he vetoed a bill that would have banned companies from making AI chatbots available to anyone under 18 unless they could ensure the technology could not engage in sexual conversations or encourage self-harm [16]. Thursday's laws leave the products available to minors and attach duties to how those products behave [3][4].

What to watch

  • Effective dates and implementing rules: the accounts of the signing give no compliance deadline for the feed restrictions or the chatbot protocols.
  • Whether anyone files the speech challenge the EFF's framing sketches, and whether a court reaches AB 1709 before platforms rebuild under-16 feeds.
  • New York's parental block on algorithmic suggestions is scheduled to take effect early next year, a nearer test of the same mechanism.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories