Build1 publisher3 min readPublished
A Llama 3.2 3B agent running offline invented a $1,990 balance on a $1,975 invoice. The open-sourced answer leaves prose to the model and hands every number to deterministic Python behind a tri-state router.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
The ordering in the graph is the load-bearing part. Deterministic code runs first, and the SLM tone evaluator is invoked only if that check passes [11]. A draft with a wrong total therefore never receives a tone score, so fluent prose cannot buy its way through on style. Numbers get settled in Python, in integer cents, matched against ledger transactions [10].
$1,990 against $1,975 is a $15 overstatement, 0.76 percent of the invoice [16]. As hallucinations go that is almost courteous, and it is also exactly the size that survives a review by eyeball. The imaginary transaction ID TX-9999 stands out immediately [4], but the fifteen dollars sitting inside a polite paragraph is the harder catch.
When the check fails, the error goes back to the drafter as a structured critique under a retry budget of N <= 2 [12]. That means one request gets at most three passes through the model before the graph stops trying and escalates to a person [17][14]. Running locally, the API bill for those retries is zero [1], so the budget is priced in wall clock rather than dollars.
The Abstain branch is the one most linear agent designs leave out. If a client has zero unpaid transactions, or the records are ambiguous, the agent refuses to draft at all [13]. That answers the failure mode the author names directly: default prompting pushes a model to guess rather than cleanly refusing or escalating [5]. Refusal has to exist as a state in the machine itself.
The evaluation layer gets the same treatment. The trap described in the write-up is teams eyeballing five outputs, or asking another LLM to rate a response from 1 to 5, without measuring statistical rater agreement or testing for length bias [6]. Quadratic Weighted Kappa is how this design answers that, calibrating the judges instead of trusting their scores [7]. Approved and edited drafts then feed continuous memory, rather than the corrections disappearing into an email client the way they do without the loop [14][15].
What would have to be true for this to move to another workload: the quantity in dispute must be computable from records you already hold. Invoicing qualifies, because a balance is checkable against ledger transactions and line items are checkable against each other [9]. Where no such function exists, the deterministic branch has nothing to assert, and the gate collapses back to a language model judging a language model, which is the thing the architecture set out to distrust [8].
One honest limit on the evidence. What the author published is a blueprint with failure modes and code patterns, not a measurement report: there is no before-and-after error rate and no reported QWK value [18]. The design reasoning holds on its own terms. The cost per approved draft is a number an adopter will have to produce themselves.
Ranked by verification strength, evidence, and original report placement.
Running a 3-billion-parameter model like Llama 3.2 locally on commodity hardware is fast, private, works completely offline and carries zero API costs.
Put in front of real operational data such as invoicing or ledger billing, a Small Language Model composed a polite email draft and in the same paragraph invented a $1,990 balance on a $1,975 invoice.
The same failure class includes fabricating an imaginary transaction ID such as TX-9999, and guessing randomly when faced with missing records.
One named failure mode is missing exit ramps: when a model is uncertain or the data is ambiguous, default prompting pushes it to guess rather than cleanly refusing (Abstain) or escalating to an expert (Escalate).
The "vibes" evaluation trap: teams evaluate by eyeball-testing 5 outputs or asking another LLM to rate a response from 1 to 5, without measuring statistical rater agreement or testing for length bias.
The author built and open-sourced closed-loop-slm-agent, a local-first agent architecture built around Llama 3.2 3B that enforces deterministic verification before any text ships, provides explicit tri-state routing, calibrates evaluation judges using Quadratic Weighted Kappa (QWK), and continuously learns from human feedback in-context.
Publishers with included, body-backed reporting in this cluster.
Follow any of these and your For You feed starts watching them — no settings page required.
build
Move execution authority out of the model and into the tool boundary1 publisher
build
The refund button is the architecture: inside the tool-use layer of a support agent1 publisher
build
Force the tool call, then hand Lightsail a long-lived key1 publisher
build
A receipt intent written before dispatch turns an SMS timeout into a poll instead of a resend1 publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Mechanism shown, outcomes unmeasured
dev.to prints the verifier's Python, so the integer-cents comparison, the invoice-existence lookup and the structural rules can be read and disputed line by line, even as the outcomes those checks produce remain unmeasured. Every performance claim around that code rests on the builder's account of his own project, and the piece reports no error rate and no kappa. The published listing also cuts off mid-function, so even the checkable part arrives incomplete.
Code published, nobody counted yet
One artefact exists: the repository went public alongside the September 2026 write-up. There is nothing else to count, no stars or forks reported and no other team saying they pointed it at a real ledger, let alone a business that bills customers through it.
The adjectives run ahead of the numbers
'Production-oriented' and 'zero hallucinated dollar figures' are heavy labels for a system with no published pass rate, and calibration by Quadratic Weighted Kappa is sold as a pillar of the design while the kappa itself never appears. Set against that, the post is candid about its own scope and leads with its own worst moment, a $1,990 balance quoted on a $1,975 invoice, rather than burying it. What inflates the claims here is the framing around the work, not anything in the architecture itself.
Author writing up his own repository
The only voice in this story is the project's sole promoter, and a clean architectural narrative is what earns attention on dev.to, which is also why the fix gets more detail than its limits. Two things restrain it: no vendor, sponsor or paid tier sits behind the post, and the code is public, so the framing can only drift so far from what the repository actually does.
Complete view of one account
There is only one source here, and it explains its own design at length, which makes it easy to read closely; whether that design actually holds up is a separate question that no one outside the author has tested, since nobody else has run the graph against a real ledger. A second account, or one engineer reporting escalation and pass rates from their own billing data, would move this assessment further than anything more the author writes.