Build1 publisher3 min readPublished
Sentinel trains its fraud model on alerted-vs-cleared cases to stop double-counting the alert itself
Sentinel's builder trained its alert model on fraud versus cleared cases, reaching 0.9465 AUC after three passes that over-called fraud on every case. The finished agent lets an LLM write the explanation and leaves every action to fixed bank rules.
The Engineer · Build desk
What happened
- Sentinel, an entry to Hacker House Goa x TigerGraph, reads six months of the Vesta card graph and returns a verdict, pattern label, actions and a SAR narrative for each alert.
- The fitted model gives the new-device flag a coefficient of -4.93, since travellers and people with new phones make up half the cleared cases.
- A deterministic policy engine encoding the bank's rulebook picks every action, and the LLM only writes prose grounded in graph queries or model coefficients.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision A team putting a model on top of an existing risk score has to choose the training population first. A model trained on fraud versus everything relearns the score's inputs and flags every alert.
- constraint The sign flips depend on one bank's cleared-case mix. A team whose cleared queue has fewer travellers or phone upgrades cannot reuse the -4.93 new-device weight.
- capability With a fixed rulebook choosing actions, a hallucinated sentence in the narrative cannot alter an approval route or the decision to write a SAR, and each action can be audited against written policy.
The over-calling comes from where the agent sits. The bank's risk score has already chosen every transaction the agent sees. Every "obvious" feature on that transaction is one the score already conditioned on, according to the post [5]. Read those features again as evidence and the alert gets counted twice. The author's first rules engine did exactly this. It restated why the alert had fired [4], and three passes in a row over-called fraud on every case [5].
The fix changed the training population. The alert model now learns fraud against cleared alerts, instead of fraud against everything [6]. Five-fold cross-validation gives an AUC of 0.9465 and a Brier score of 0.0927 [7]. Those figures measure separation inside the alerted population of one dataset, six months of the Vesta card graph [1][7]. The post does not report precision after the retrain. The only precision figure is the roughly 50% the author could not beat by intuition before it [12].
Three coefficients explain that ceiling. The new-device flag, id_15 = 'New', carries -4.93. Half the cleared cases in the labelled history are travellers or people with a new phone, and both trip the flag [8]. Amounts over $1,000 carry roughly -0.6, because fraud in this data clusters at $30 to $100 online [9]. The bank's risk-score decile gets zero weight on risk_score alerts [10]. "The decile is why the alert fired. Using it as evidence would be evidence-laundering," the author wrote [11].
For those signs to transfer, another bank's cleared queue would need a similar share of travellers and new phones [8]. Its score would also have to condition on the same inputs.
The action path is kept away from the model. Every action comes from a deterministic policy engine encoding the bank's rulebook. Everything the LLM writes is grounded in a graph query or a fitted coefficient [3]. "The LLM writes prose; the policy engine decides. Two jobs, and only one of them can hallucinate," the author wrote [2]. I think this is the right split for a workflow whose outputs include approval routes and a SAR narrative [1]. A reviewer can check each action against the rulebook and each sentence against the query behind it.
The graph work shows the same care. The hand-written weakly-connected-component query, ring_wcc, runs a BFS fixpoint over the card-device projection and skips hub devices above a degree cap [13]. At the default cap of 100 it returned about 5,500 cards across 9,200 devices in five iterations. At cap 3 it still returned 1,235 [14]. Cutting the cap from 100 to 3 shrank the component by about 78% [2], and one seed still reached a third to half of all cards [15]. I would copy what came next. The author kept WCC size in the ledger as blast-radius context and set the ring flag from a narrower peer filter [16].
Eighteen GSQL queries are installed. The eight that fire per investigation go out through asyncio.gather on an httpx.AsyncClient. Wall-clock fell from about 25 seconds to about two [17], a 12.5x gain [1]. The speedup helps the analyst waiting on one alert. At twenty investigations a month [20], the sequential version would have used about eight minutes of query time in total [3].
What to watch
- Precision figures for the fraud-vs-cleared model, since the only precision number in the post is the roughly 50% ceiling reached before the retrain.
- Whether the coefficients keep their signs on a second bank's cleared-case history, especially the -4.93 on id_15 = 'New'.
- The per-call latency of the default SENTINEL_GRAPH_VIA_MCP=1 path, which sends every graph call through tigergraph-mcp over stdio.