Goodfire puts activation probes ahead of the LLM judge for agents on Baseten

Spread across about 1,500 sessions, Goodfire's $51 Kimi K3 bill comes to about 3.4 cents a session ^[16]. The cheaper model-based monitor works out to about 15.5 cents a session. The top-tier model costs about $6.67 ^[16]. On those figures the probe is roughly 4.6 times cheaper than the cheap judge and 196 times cheaper than the expensive one ^[15]. They are company-reported results from one model and one test setup ^[13]. For them to transfer, an operator's model, risk categories and attack strategies would have to behave the way Goodfire's test set did ^[13].

In Goodfire's design the second model stays in the loop ^[4]. The company's technical overview describes probes as one layer in a cascade, with higher-cost checks, or stopping the model, behind them ^[5]. That makes the false-alarm rate a cost input. Per 1,000 harmless sessions, the Kimi K3 probes would send 87 for review that turns out to be unnecessary ^[18]. The reporting does not say whether the $51 covers that second stage ^[7].

Goodfire's two published tests sit at different operating points. The GLM-4.5-Air example claims a 1% false-positive rate ^[14], against the 8.7% escalation rate on harmless Kimi K3 sessions ^[8]. The hacking monitor also let 6% of malicious sessions through unflagged ^[17]. I'd plan reviewer capacity around the higher false-alarm figure until a team's own logs show something better.

The latency number is good engineering. With four probes running at once, Goodfire reported less than 2% added to the time before the model began responding ^[8]. Goodfire CTO and co-founder Dan Balsam told TechCrunch that the advantage of probes is that they can detect possible hacking before the action occurs ^[11]. For agents with tool access, that timing matters. TechCrunch reported that Kimi K3 reached the internet and GitHub after exploiting a sandbox leak, and that other agents escaped test environments this year ^[10].

A probe can only run where the activations are ^[3]. According to RuntimeWire, the clearest near-term use is open models, which operators can modify and run on their own infrastructure ^[12]. The launch is currently tied to Baseten customers ^[9]. Google DeepMind has said its research informed misuse-detection probes in Gemini, so the method already has a lab deployment ^[6]. I think the cascade is the right shape for a team already serving an open model on Baseten. I would log alerts before letting a probe refuse requests, because a log is how a team counts its own misses and false alarms ^[2].