Build1 publisher3 min readPublished
Flock Safety's fake police account searched live cameras in Dunwoody and Bryan
Audit logs obtained by a Dunwoody resident show Flock Safety staff running FreeForm queries for masks, signs and religious symbols against production camera networks, with explicit permission on record for one of the jurisdictions involved.
The Engineer · Build desk
What happened
- Audit logs obtained by a Dunwoody resident and published by 404 Media on September 17th show accounts named Flock City PD searching live camera networks in two named cities and several unidentified jurisdictions.
- The queries covered people wearing masks, crowds, groups holding signs, political bumper stickers and religious symbols captured by real cameras.
- Flock Safety told 404 Media the searches were employees testing FreeForm moderation and demonstrating queries police should avoid, and were not customer investigations.
- Some prompts were blocked and others produced warnings, while searches for "crowd", "person wearing scrubs" and "large group with signs" were allowed to run.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Because a First Amendment-adjacent query executes with a warning attached, a buyer has to look past the block list to assess the guardrails; enforcement only begins after the images come back.
- exposure Anyone matched by one of these queries is a resident of a city whose cameras answered it, and the match happened with no investigation open on them.
- decision Agencies running FreeForm have to settle in the contract whether vendor staff may query their cameras at all, and under which account.
- precedent Consent collected city by city sets the floor for the next round of live testing, since the permission on record covers Dunwoody alone.
A moderation test lands on live cameras because of what it is testing. Flock Safety said on April 16th that offensive prompts and searches using personal information should return no results, and that queries which may implicate First Amendment rights instead produce a warning, can proceed, and get referred for audit [7]. The first half of that is a claim about what a pipeline returns. Run a blocked prompt against a synthetic image set and a clean empty result tells you the fixtures had nothing to match. Run it against Dunwoody's cameras and the same empty result means the filter held on a corpus that did contain people. Flock Safety has argued that real-world testing is necessary to train visual models and verify moderation [13].
The constraint is real. I just don't think the conclusion follows. A recorded corpus, licensed once from a consenting jurisdiction and replayed, gives the same signal on blocked and allowed prompts that a live query gives. It will not give you a fresh match against this morning's traffic. Whether you need that depends on whether the thing under test is the filter or the recognition model behind it.
The warning tier is where the design gets thin. By Flock Safety's own account a warning is an accountability step with no technical prohibition behind it: it works if the user stops, if the search is sent for audit, and if an agency enforces something afterwards [8]. All three of the prompts the report identifies by name ran: "crowd", "person wearing scrubs" and "large group with signs" [4][16].
FreeForm takes a description in ordinary language, including clothing, visible accessories and vehicle details [9]. A plate query asks whether a known identifier passed a camera; a FreeForm query asks an image-recognition model which unknown people or vehicles resemble a description [10]. Flock Safety says people searches use observable characteristics and not facial recognition [11]. That distinction is a fact about the model, and it changes nothing about the subject, because "large group with signs" still resolves to particular people standing in a particular street [4].
Dunwoody is the one jurisdiction on record. Flock Safety said in April that the city had given explicit permission to participate in its testing program [5]. The same logs cover Bryan, Texas, and several unidentified jurisdictions [1], and no permission from any of them is on record [15]. When the searches ran is also unknown [12]. Naming the fictitious tenant "Flock City PD" at least made it easy to grep for afterwards.
Flock Safety compared the process to cybersecurity testing, where a simulated attack is used to check whether a defense holds [6]. In a penetration test the owner of the asset scopes the engagement and agrees to the damage. The people matched by these searches do not own the cameras. Garrett Langley built the company around the promise that software, audit logs and local oversight would keep the resulting surveillance accountable [17], and the log is how a resident, Jason Hunyar, found out [1].
Separation for this class of product has to sit below the account. A tenant that can address production imagery is production, whatever label it carries in the interface, and the control that decides the question is which image store the query can reach. Flock Safety started by turning passing vehicles into structured evidence, plates, colors, body types and visible damage, and now sells video cameras, gunshot detection, drones and cloud software that connects data across participating organizations [14].
What to watch
- Whether Bryan, Texas, or any of the unidentified jurisdictions confirm they authorised testing on their camera networks.
- Whether Flock Safety publishes the date range of the Flock City PD searches, which the 404 Media report leaves open.
- Whether the moderation ladder changes so that First Amendment-adjacent prompts block instead of warning and proceeding.