A SIGMOD 2027 benchmark scores the SQL a role was never permitted to run

The sharpness is a scoring effect, according to the dev.to write-up: a metric that never looked at authorisation had been counting violations as successes, and applying a policy moves those queries out of the success column ^[11], while the SQL-writing task itself is no harder than before ^[11].

The reason a well-built stack produces such queries is the order of operations. Every NL2SQL stack narrows the schema before the model writes anything, because a real schema runs to hundreds or thousands of objects and will not fit in a prompt ^[12]. The narrowing is done by a ranker matching the question's words against a catalogue, with no idea who is asking, while the database's grants, row-level security and VPD policies act later, at execution ^[13].

So a support agent's question scores near hr_compensation, the model writes correct SQL against the table it was shown, row-level security filters every row, and the user gets "No records found" ^[14]. Nothing errored, and nothing was logged as a denial ^[14]. The agent then states that there are no compensation records matching the question, and the person reading it cannot separate "the data doesn't exist" from "you aren't allowed to see it" ^[15].

A schema object called hr_compensation_2027_layoffs tells a reader something real even when zero rows come back, and row-level security cannot retract it, because the disclosure was in the DDL ^[16].

Granularity is where adoption costs something. The augmented set covers 3,353 columns across 399 tables ^[5], which is about 8.4 columns per table, so the surface a column-operation policy has to describe is roughly eight times the count a table-level grant does ^[1].

The paper says the pipeline, toolkit and datasets are coming to a public repository ^[18], and the post's author writes that he has not run the benchmark himself ^[19]. The published result is one sentence: many high-performing systems, open-weight LLMs especially, "show sharp performance degradation once access constraints are in place, due to frequent RBAC violations" ^[9]. For that to predict what happens to your own numbers, your grants would have to be column-granular and your roles would have to resemble the ones the paper's LLM-assisted pipeline synthesised per database, scoped administrator roles included ^[8].

Re-checking permissions after the SQL comes back does not cover the disclosure. The table name has already gone into the prompt, and the check can only report that the query was disallowed ^[20].