Skip to content

Build1 publisher3 min readPublished

Checking that a cited receipt ID exists turns finance-agent quality into a regression test

A dev.to walkthrough makes the shippable unit a five-field decision contract with evidence IDs, scored by a harness of 50 to 100 finance cases in which three of the four groups exist to see whether the agent stops.

The Engineer · Build desk

Illustration accompanying Checking that a cited receipt ID exists turns finance-agent quality into a regression test

What happened

  • The walkthrough's decision contract for a transaction-review agent has five fields: action, confidence, evidence_ids listing bank_tx_1842 and receipt_883, needs_review, and a reason string.
  • It recommends a starter dataset of 50 to 100 finance cases split into routine, missing-evidence, conflicting and adversarial groups.
  • Scoring is split across five layers: schema validity, evidence grounding, policy compliance, decision quality and operational behaviour.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • decision Fixing what the agent may decide before you pick a model or framework puts the propose / request review / execute boundary with whoever owns the chart of accounts, not with whoever writes the prompt.
  • cost Someone has to author and label 50 to 100 finance cases, including the conflicting ones where the vendor name and the attached document point to different categories, and that labelling work falls on accountants.
  • constraint Once refusal is a stored expectation, a prompt tweak that improves demo answers can fail the build. That closes the route where better-sounding output ships without review.
  • capability Event-level traces make retry safety and timeout handling testable before release, so duplicate reconciliation records become a failing case.

Evidence grounding is the layer that needs no model. The starter checks in the dev.to walkthrough are deterministic: valid IDs, matching account IDs, date-window constraints, amount consistency [11]. Those resolve against systems you already run. If receipt_883 does not exist in the document store, the case fails before anyone reads the reason string [4].

Whether the evidence is relevant is a separate question, and the harness is asked both: that every cited document exists, and that it supports the claim [11]. Date window and amount match are only proxies for relevance.

The refusal path is what makes the contract testable. In the missing-receipt case the expected action is request_evidence, allowed_categories is empty, must_cite is empty, and must_not lists post_journal_entry and mark_reconciled [8]. Two of those fields are negative expectations. The case fails if the agent posts the entry, whatever category it picked, and the post wants that failure to surface in the build the moment a prompt or model change causes it [9].

One layer up, the scorer separates propose, request review and execute, on the argument that a suggestion can be acceptable when an automatic journal entry is not [12]. Grading that distinction means something only when the tool layer enforces it; a scorer reading final_action while the agent holds a write credential is measuring intent. Decision quality is kept apart from writing quality, and the author wrote: "A beautifully explained wrong category is still wrong." [13]

Each tool call is logged as an event, which is where the duplicate-webhook case shows its shape: step 3 calls create_review_task and gets task_551, step 4 calls it again and gets already_exists, and final_action is request_review [15]. From a trace like that you can test that a repeated event produces one task and that a failed document lookup does not silently become "no document". You can also test that a timeout appears as an exception state and that writes carry an idempotency key [16].

I would push back on the sizing. Fifty to a hundred cases across four groups [6] is 12 to 25 cases per group on an even split [18]. The adversarial group covers prompt injection inside a memo, unusual currencies, duplicate webhooks and transactions outside the agent's authority [7]. At the low end that is about three cases per failure mode [22]. Three cases catch a regression you have already seen once, but they do not establish that an injection class is handled.

For the grounding layer to transfer to your stack, your evidence needs IDs a test can resolve. Bank transactions usually have them. Receipts dropped into a blob store under generated filenames often do not, and then evidence grounding degrades to checking that a string is present. The post does not report what any model scored against these four groups; it defines the contract, the dataset shape and the five scoring layers [10], and says to fix the decision boundary before choosing an LLM or tool framework [20].

What to watch

  • Whether the author publishes per-layer pass rates for a named model against the 50-to-100 case set.
  • Whether the four case groups ship as a shareable dataset with the must_not lists intact.
  • Whether agent tool frameworks start attaching idempotency keys to write calls by default, since the trace tests assume they exist.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories