Valid JSON, Wrong Bucket: Why A Model Answer Is A Proposal, Not A Result

A crash triage queue stopped draining at 02:40 while every instrument reported health: the worker process was alive, the HTTP client saw 200 OK, and the JSON decoder returned objects ^[1]. What had actually failed was the one thing no schema checks, because a heap corruption report had been filed into the network timeout bucket with a plausible label and valid JSON around it ^[2].

The account comes from a write-up on dev.to, and it carries a disclosure: the article was prepared as part of MonkeyCode's product outreach, with MonkeyCode's free model access and free server option appearing in the pipeline as the proposal source and a shadow voter ^[8]. The vendor placement is what it is. The failure mode is general enough to be worth separating from it.

The service sent raw crash stacks to a free model endpoint and got back a small set of triage labels ^[3]. The model produced clean JSON every call and the schema stayed stable; the values drifted ^[3]. On one day heap_use_after_free came back as network_timeout, on another stack_overflow came back as plugin_crash ^[4]. There was no parse error and no timeout, and retries made things worse because each attempt returned another valid, equally confident answer ^[5]. A payload carrying "label": "network_timeout" and "confidence": 0.96 satisfies every serialization check, so nothing at the transport layer can see the bug ^[6].

The response described is a C++ sidecar that treats the model label as a proposal, applies a deterministic contract first, records disagreement in a drift ledger, and routes unstable labels to quarantine ^[7]. The contract is deliberately small: it fires only when a normalized stack contains an exact needle, and its stated purpose is a stable reference point for drift detection rather than coverage ^[9]. Four rules: "heap-use-after-free", "stack-overflow", "etimedout" and "null pointer dereference", mapped to their buckets ^[10]. Normalization lowercases printable characters and collapses whitespace and non-printable runs to a single space ^[11]. When the contract and the model disagree, the sidecar does not retry; it increments a ledger cell keyed on the contract bucket, and agreement is recorded too ^[13]. The window slides, so old entries are dropped and the rate is disagreements over entries in the window ^[14].

Two limits are visible in the code as published. ContractVote returns nullopt when no needle matches ^[12], and ledger cells are keyed by the contract bucket ^[13], so a wrong label on a stack the contract does not recognise generates no entry at all; drift outside those four signatures is unobserved ^[2]. Of the four labels named in the incidents, plugin_crash has no contract rule ^[1]. And DriftRate returns 0.0 both when a bucket is missing or empty and when the window is clean ^[15], which makes "no data" and "no disagreement" the same number on a dashboard ^[3].

Worth watching: whether the quarantine path is specified beyond the naming, since the excerpt is truncated where it begins describing the ledger as reactive and non-blocking ^[16]; whether contract coverage grows as new signatures are hand-verified ^[9]; and whether teams copying this pattern log the disagreement rate per bucket rather than a single global number ^[14].