Skip to content

Build1 publisher3 min readPublished

A three-entry allowlist test fails the build when this refund agent's rules import a framework

Part 14 of a dev.to series describes what its author calls "a support agent that can work out whether you're owed a refund, and cannot give you one", with the eligibility rules pinned to the JDK by a test.

The Engineer · Build desk

What happened

  • In the fourteenth post of a dev.to series, the model handles intent and retrieval while software scopes every lookup to the logged-in user, decides eligibility and assigns a risk tier.
  • A gate reads that tier: LOW actions run, MEDIUM to HIGH actions wait for a human to approve, and VERY HIGH actions are only ever proposed for a person to execute.
  • Nothing rate-limits proposals, so the post says a patient adversary could bury a reviewer in plausible-looking refund requests until approving becomes reflex.
  • The agent has never served a real customer, and every scenario in its eval suite was invented by the author.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • capability Because money only moves after a person approves, a model that is wrong about eligibility costs a wasted review instead of a payout.
  • exposure Whoever staffs approvals inherits the risk: with no rate limit on proposals, the only defence against a flood of plausible ones is a reviewer's attention.
  • decision Anyone adopting this writes the durable queue and audit adapter themselves; what is reusable is the port contract naming durability, notification and permanence.
  • constraint Fixed risk tiers are what keep the deterministic side simple, so the first customer who needs different tiers forces a rewrite of where the authority to decide lives.

The test that keeps the business rules clean is three strings long. Domain classes may reference `dev.tonal.support.domain..`, `java..` and `javax..` [3]. Two of the three are JDK namespaces, which leaves exactly one application package reachable from the rules that decide eligibility [4]. It is an allowlist because the alternative, enumerating forbidden layers, needs an edit every time a package is added [5]. Add a JSON library to a policy class and the build goes red, and the author says that was confirmed by breaking it on purpose and watching it fail [6].

Spending a test on something that dull is a bet about decay. According to the post, nobody announces the afternoon they import a framework into the business rules, and a few months later "the AI doesn't decide business questions" is not true any more [7]. "A sentence in a README can't catch that. A failing build can," the author wrote [8].

The approval queue and the audit trail are ports with in-memory adapters behind them, and the ports already state what a real implementation owes: durability, notification, permanence [9]. An in-memory audit trail is durable until the process restarts. The author puts writing the real adapter at an afternoon and says deciding what it must guarantee was the part worth the thinking [10].

Two omissions are labelled decisions rather than backlog. Multi-tenancy is out because per-tenant risk policies turn the tier lookup into runtime configuration, and the author wants evidence that a second tenant genuinely needs different tiers before letting a form decide who gets to decide [11]. LLM-as-judge stays out of the eval suite because a model grading a model has its own variance and would need its own eval first [12].

Whether the shape transfers depends on three properties of the system you would put it in. The decision has to be expressible in deterministic code over JDK types, or the allowlist is the first thing you delete. The tiers have to be fixed per deployment, for the reason the multi-tenancy deferral gives. And the execution path has to be idempotent, because in this build an approved refund executed twice after a retry is not prevented [13].

The eval numbers reproduce because the suite grades deterministic classifiers, so a clone returns the author's exact figures, and the whole project runs with no API key [18][19]. Nothing in it notices if the knowledge base quietly gets worse while every test stays green, which the post files as Planned [15]. Every scenario in the suite was invented by the author, which the post calls good at catching regressions and useless against the thing that was not thought of [16]. "Real users are more imaginative than I am, and they aren't trying to be fair," the author wrote [17].

What to watch

  • Whether the retry double-execution and the missing proposal rate limit move from Planned to shipped, since both land on the human approval path.
  • Whether a durable adapter appears behind the approval-queue and audit ports, and what it guarantees against the three the ports name.
  • Whether any run against real customer traffic gets published, given that every graded scenario so far was written by the author.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories