Product1 publisher2 min readPublished
Waymo called police on its own riders after cabin cameras flagged a firearm
Two teenagers were arrested in early September after the robotaxi pulled over and alerted emergency services. Waymo described the policy in an email, and left out the confidence threshold and the human review step.
The Product Desk · Product desk
What happened
- Waymo said it detected a violation of its terms of service involving a firearm, pulled the car over and alerted emergency services in early September, according to the Los Angeles Times.
- Officers arrested the two teenage passengers after allegedly finding a loaded AR-style ghost gun inside the vehicle.
- Waymo did not explain how the system decides something may be a firearm, how confident it must be to flag it, what humans do between alert and call, or which other violations trigger one.
- Five Waymos were vandalized and set on fire during anti-ICE protests in Los Angeles, where whether the cars were capturing images of protesters was unclear.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- precedent Every product team shipping a camera into a car or a home now has a worked example of a vendor deciding on its own to escalate what it sees to police, and customers will start asking for that policy in writing.
- exposure A false match costs the rider: Koopman's point about accuracy means some flagged objects will not be guns, and the failure mode is a police response to a stopped vehicle.
- constraint A rider cannot estimate how often footage of them leaves the company, because the request and compliance figures are not published. The one choice a rider can make with information is whether to get in the car.
- decision Operators have to pick between enumerating the triggers and thresholds in the rider terms, or defending an emailed "we may reach out to local law enforcement" after the first arrest.
Between a camera frame and a phone call sits a classifier, and then, probably, a person. Bob Somers, a former engineer at the autonomous vehicle company Zoox and a technical adviser to the AV-safety nonprofit SAVE-US, told The Verge he would expect Waymo's system to look for a range of potentially dangerous weapons and behaviors, and said the software may not need to decide with certainty that it has seen a firearm: it could flag something that resembles one for further review [12]. "If there is a certain set of objects detected, then that would probably loop in a remote operator to take a look at what's going on," Somers said [13].
Phil Koopman, a Carnegie Mellon University professor emeritus who researches autonomous vehicle safety, said computer vision systems can be trained to recognize objects and potentially dangerous situations [14]. "If you can train a computer to look at the road and decide how to drive, you can train a computer to decide if somebody's holding a gun," Koopman said, adding that those systems are "never 100 percent accurate" [15].
Earlier cases ran the other direction. Law enforcement agencies have obtained search warrants for footage captured by Waymo's cars on a number of occasions, according to Bloomberg, and the LAPD last year obtained footage of a hit-and-run recorded by a nearby Waymo [4][5]. In the September stop, nobody came asking: the car flagged something inside itself and the company made contact [11]. Waymo does not publicly disclose how many law enforcement requests it receives or how often it complies [7].
The closest comparison is Flock, whose automated license-plate reader network has given police the ability to track cars across jurisdictions and prompted fears about government surveillance and misuse [16]. Flock's cameras are fixed. Waymos drive through cities with cameras pointed outward and inward [16].
For anyone putting a camera in a space a customer treats as private, two things matter: whether the system only records or classifies in real time, and whether the operator hands material over only under legal process or contacts police on its own. Waymo is on the record on both. Spokesperson Julia Ilina said the company uses its internal cameras to help enforce its terms of service, including detecting prohibited items and behavior, with automated technology helping its teams identify potential violations such as firearms inside the vehicle [8]. "If Waymo detects a firearm in the vehicle, we may reach out to local law enforcement," Ilina said [9]. The verb in that sentence is "may".
What to watch
- Whether Waymo begins publishing counts of law enforcement requests, self-initiated contacts and compliance rates.
- Whether a false match becomes public, and what the rider is told about the object the system flagged.
- Whether Waymo's rider terms come to enumerate which violations can prompt a call to police.