Build1 publisher2 min readPublished
A fraud agent built on TigerGraph keeps its verdicts in code that fails the build on any LLM import
Fraudagent, a TigerGraph challenge entry, settles 20 fraud alerts in code whose build fails on any LLM import. Its verdicts match bit for bit with no API key, so the calibration bug its authors blame for clearing fraud lives in code a test can reach.
The Engineer · Build desk

What happened
- TigerGraph's Hacker House Goa challenge supplies 590,742 card transactions from the IEEE-CIS dataset with the fraud label removed.
- Entrants get 5,565 closed investigations with analyst notes in place of labels, plus 20 new alerts to decide.
- The agent's decision code sits in a core package guarded by a test that parses each file and fails the build on any LLM library import.
- Run with no API key, the agent produces bit-identical verdicts, probabilities, actions and approval routes, and only the wording changes.
- The builders say two measurement mistakes in calibration silently inverted the agent until it waved fraud through.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint An entry that answers in one pass cannot fill the before-and-after fields honestly, so the challenge format pushes builders toward a loop with an explicit stopping rule.
- capability Reviewers can rerun all twenty answer files offline and diff them, so a scoring regression shows up as a changed verdict file at no model cost.
- exposure The SAR narrative a regulator would read is the model-written part, so it is the one output that can differ between runs.
- decision Teams building regulated agents get a pattern they can copy: put the audited path in one package and let an import check in the build keep model calls out.
Two lines in the answer format set the architecture. Each alert's next best action and approval route must be recorded before any additional evidence is requested, and again after it is received [4]. The builders argue that a single-shot pipeline cannot produce that pair honestly [13]. So the LangGraph agent commits to a provisional action and route, picks the question worth asking, then records a revised action and what changed [5]. One conditional edge chooses between gathering more evidence and acting [5].
The model sits outside that choice. The Bayesian ledger, policy engine, approval routing, stopping rule and episode reconstruction all live in core/ [7]. The model picks which graph primitives to run and writes the prose, including the case summary and the SAR narrative [7]. If it proposes an action anyway, the policy decision object drops it before it reaches an answer file [7]. The build test that guards the boundary ends in one assertion [6]:
```python assert not offenders, f"core/ must stay LLM-free: {offenders}" ```
This is good engineering. A reviewer does not have to trust the diagram, because crossing the line breaks the build. "The practical consequence, which I like more than the principle: the agent runs end to end with no API key at all," the author wrote [12].
The available text of the write-up stops inside the retrieval query, before either measurement mistake is described. The architecture still narrows where they could have been. Probabilities and verdicts come out the same with or without a model [8], so whatever inverted them sat in deterministic code or the data it reads, not in a prompt. I'd check the bank's risk score first. It is the one signal every transaction carries [2]. The dataset README is blunt about it: "Above 0.7, most flagged transactions turn out to be legitimate. Some fraud scores near zero." [2] A scorer that treats that number as a stand-in label rewards an agent for agreeing with false alarms.
Case memory has its own version of the trap. Vector search alone returns prior cases that sound alike. Because the narratives are templated, it returns five card-testing cases when asked about a different burst pattern [9]. Traversal alone returns cases that share a card, device or customer [9]. The author called that "Precise, and usually empty." [14] The fused GSQL query pulls k * 8 vector candidates, keeps only cases opened before the as-of time, and adds closed cases linked through the same device [10]. Without that as-of filter, a replayed benchmark could retrieve an investigation opened after the alert under review.
What to watch
- The rest of the write-up naming the two measurement mistakes, and whether either used the bank's risk score as a label.
- Whether the twenty reproduced answer files change after the calibration fix, and by how many verdicts.
- How the challenge judges score investigations when the dataset ships with no fraud label to grade against.