New York City Council proposes making AI vendors liable for harm from jailbroken tools

Outside validation and a kill switch are costs a vendor can estimate before it sells a system in the city: pay the validator, build the switch ^[1]. The jailbreak suit is harder to price ^[2]. A jailbreak is a user getting a tool past the limits its maker set, so the vendor's exposure depends on what buyers do after the sale. Better engineering lowers that exposure without capping it. Fortune's account does not give damages amounts, the size of whistleblower payments or an effective date, so none of this can yet be put in dollars.

The duty attaches to systems sold in the city ^[1], and the companies expected to carry it have also put down roots there. Google has more than 14,000 employees in New York City, and Meta leases 1.2 million square feet at 50 Hudson Yards ^[12]. Anthropic leased an entire 16-story building at 330 Hudson Street this summer and expects more than 1,000 city staff by year's end ^[13]. Google and Anthropic alone would then employ more than 15,000 people in the city ^[14]. Moving those desks (or rather, threatening to) would not end an obligation tied to sales. "We obviously are not in any way looking to stifle innovation. We want to ensure that New York stays the AI capital of the world, but with that comes responsibility," Menin told Fortune ^[6].

Washington has two routes to end this. Congress's own output so far is a 2017 bill that went nowhere and nine closed-door Senate forums that produced a roadmap and no bill ^[16]. Preemption by statute has already failed once: the Senate stripped Sen. Ted Cruz's 10-year ban on state AI laws by 99-1 in July 2025 ^[8]. The Cruz-Klobuchar-Thune bill is the second attempt ^[10]. Litigation has worked faster. The Justice Department task force President Trump created to sue states over AI laws made Colorado its first target, and Colorado gutted its own law five weeks after the suit was filed ^[8][9].

The third path is that the package survives and gets copied. Menin described the bills as a case study from a Columbia course she taught, "When Cities Take the Lead," on what cities should do when the federal government is deregulating or failing to act ^[15]. "This is perhaps one of the greatest cases of that, where you have a product in AI that has very little regulation," she said ^[7]. OpenAI has made a version of the same argument one level up, writing in a July post that states could fill the gap by passing laws that mirror one another, in place of a federal one ^[11].

I think the jailbreak suit is the provision most likely to be narrowed, either before passage or after a federal challenge. It is the one cost a vendor cannot cap through its own engineering, and Colorado retreated within five weeks of being sued ^[9]. The counter-case is the footprint. Two companies with more than 15,000 staff between them in one city ^[14] have more reason to bargain with its council than to fight it in court. The view is wrong if the package passes with the right to sue intact and the vendors respond by building compliance programs instead of suing.