An audit of 120 published agent outputs turned up exactly one commitment

Each commitment COGEXT emits is a record with an action, an object, a recipient, a deadline, a class, a confidence number and a verifier query. The highest-risk example in the audit came from the anthropics/anthropic-cookbook repository. The promise reads "I will redeploy the api-server to apply the changes": action redeploy, object api-server, recipient NONE, deadline NONE, and the verifier query "check docker-compose logs or container list for api-server recreated with new configuration" ^[14]. That query is specific enough to run against a host, but the deadline field that would say when to run it is empty.

The author treats the distance between confidence and verifiability as the audit's sharpest finding. Mean confidence across the 13 commitments was 0.79 ^[13]. Twelve of the 13 do not include a time reference ^[19]. Confidence answers whether a promise was made; it does not answer whether the promise can be checked. COGEXT will not mark a commitment fulfilled until external evidence confirms it, so the audit reports all 13 as unverified at extraction time ^[12].

The 0.8% figure is a statement about a specific corpus ^[4]. A notebook cell in a cookbook repo is a demo someone chose to commit, and the extractor only recognises specific, first-person, future statements, which the author flags as "a property of the extractor worth knowing" ^[6]. The other 119 samples were narrative, tool traces, code or past-tense reports ^[5]. For 0.8% to transfer to your own traces, your agent's output would have to be distributed like published demo output. Your working definition of a commitment would have to be as narrow as COGEXT's. A planner that logs that it will retry in 30 seconds is a first-person future commitment with a deadline attached, and it would score differently on both counts.

The premise came from somewhere else. The author wrote that he built COGEXT because "8 production engineers on Reddit all described the same problem: AI agents make promises, and there is no system that tracks whether those promises are kept" ^[17]. This audit ran on published output instead.

Thirteen items make coarse percentages. The 61.5% with no recipient is 8 of 13 ^[18], the 92.3% with no deadline is 12 of 13 ^[19], and one reclassified sample moves either figure by about 7.7 points ^[20]. Stratum B was also selected on the strings "I will" and "I'll" ^[7]. The author says of it: "This is not a prevalence estimate" ^[7].

The open-ended promises the audit quotes include "I will redeploy the api-server", "I will open a pull request" and "I will aim for a window seat" ^[22]. A single confidence score does not distinguish a redeploy from a seat preference. All 150 POSTs to the /api/v1/ingest endpoint succeeded at a one-second delay between calls ^[15], and the raw files are published as agent_samples.csv, audit_results.csv and commitments_all.csv, with a public audit URL per commitment ^[16].