The all-clear is the bug: how to make an AI RLS audit repeatable

A developer asked an AI coding agent to audit the row level security on a Supabase project, got three findings plus an all-clear on everything else, ran it again, and got two different findings plus the same all-clear ^[1]. The useful part is the response: rewrite the prompt so the model never chooses what to inspect, hand it one fixed SQL statement against Postgres catalogue views, and leave it only the job of explaining the rows that come back ^[3][5].

The non-determinism in the findings is a nuisance. The defect is the all-clear. As the dev.to author puts it, an agent asked to go look enumerates whatever it happens to notice and then phrases everything it never checked as a pass, so the clean bill of health carries no information ^[2]. Two runs, two answers, both ending in "the rest is fine," is a system reporting coverage it does not have.

The replacement is a single statement with five checks, unioned and ranked by severity ^[7][8]. Tables in `public` with `rowsecurity` false, rated CRITICAL: readable by anyone holding the anon key, which ships in the frontend bundle by design ^[7][12]. Policies where `qual = 'true'`, also CRITICAL: RLS is on, a policy exists, and every row goes to everybody, which is why this passes any check that only asks whether RLS is enabled, including most dashboard checkmarks and most agents ^[7][13]. `with_check = 'true'`, rated HIGH, lets a user reassign a record they own to someone else, and according to the author is almost always added to silence an insert error rather than to grant anything ^[7][14]. Public storage buckets, also HIGH: fine for avatars, wrong for the invoices and scanned documents that end up in the same bucket because public buckets are where uploads work on the first try ^[7][15]. Last, REVIEW: RLS enabled with no policy attached, not a breach, but the reason a feature quietly returns an empty list instead of an error, and the usual prelude to someone disabling RLS to fix it ^[7][16].

Two design details matter more than the check list. First, read-only is a property of the SQL rather than an instruction the model is trusted to honour: every query reads `pg_tables`, `pg_policies`, or `storage.buckets`, nothing selects from an application table, and nothing writes ^[5][6]. The prompt also forbids the model from running `ALTER`, `CREATE POLICY`, `DROP`, or a migration, and asks for fixes described in words ^[10]. Second, the block runs standalone in the Supabase SQL editor with no agent involved ^[17], which means the model contributes nothing at detection time ^[19].

Two honest limits. The permissive-policy checks are equality tests against the literal text `true`, so a policy that is effectively open by some longer expression does not appear in the output ^[20]. And the prompt's own step 5 asks the model to list which findings could silently return after a future migration or prompt, and to explain why a one-time audit cannot catch that ^[11]; the author calls this the interesting step ^[18]. That is an admission the artefact is a snapshot.

What to watch: whether this pattern moves from a paste-in prompt hosted at defencecore.com/audit ^[4] into a migration gate, where the same five checks run on every deploy and fail the build. Until then the honest reading of a clean result is that five specific conditions were absent at one moment, which is a much smaller claim than "your RLS is fine" and considerably more useful.