A guard's JSON competence predicted its mid-trajectory catch rate better than its jailbreak score

A guard inspecting a trajectory is not reading prose. It is reading a serialized record of a plan: the user query, the tool definitions the agent was handed, the arguments it filled in, and what came back. The TraceSafe authors describe the detection task as pinpointing subtle contradictions and execution errors distributed across exactly those layers ^[16]. That is a parsing job before it is a safety judgement. To notice that an argument contradicts the schema the tool declared, you have to hold both structures in the same frame and keep the nesting straight. So the first finding reads as mechanical rather than surprising once you state it that way: efficacy tracked structural data competence, JSON parsing among it, more than semantic safety alignment ^[1].

The correlation is where I wanted a number. The abstract asserts a strong correlation with structured-to-text benchmarks and near-zero correlation with standard jailbreak robustness ^[2]. In the v1 HTML, the coefficient sits inside an empty pair of parentheses ^[6], which is the strongest claim in the paper and currently also the least legible one ^[17].

Twenty guard systems ^[18] over more than a thousand instances ^[4] is a real sample, and the construction method is honest about its own bias. Risks were injected by Benign-to-Harmful Editing, deterministic edits to natural trajectories that keep the planning logic intact and yield step-level ground truth ^[12]. The authors chose that over free-form harmful generation, which they say produces artificial behaviour, and over hand annotation, which they call prohibitively labour-intensive ^[13]. Both objections are correct, and the choice still shapes the result. For this ranking to transfer to your stack, three things would have to hold: your failures look like plausible-but-wrong arguments emitted by the model, your guard sees the whole trace rather than a sliding window, and your tool schemas are as explicit as the benchmark's. If your real risk arrives as text smuggled back through a tool return and never contradicts a schema at all, a structural-competence ranking tells you less.

The temporal result cuts in a direction worth sitting with. Accuracy held up as traces got longer, and later steps were easier because the model could stop reasoning from static tool definitions and start reasoning from observed execution behaviour ^[8]. Invert that. The first call is the hardest one to judge, because there is no behaviour yet, only declarations. It is also the only place a gate can stop anything, since intermediate steps can do damage while the final answer looks clean ^[10].

Which is why the paper's closing recommendation, joint optimisation of structural reasoning and safety alignment ^[15], is not a hedge. A guard that is well aligned and cannot parse a nested call will wave through the trace it was bought to flag. TraceSafe-Bench is a preprint, and its own framing is that no fixed-trace, localised-ground-truth benchmark existed before it ^[11]. Treat the leaderboard as provisional and the mechanism as the part you can act on.