Telling an agent's harness bug from a model bug takes a replay of the whole run

An agent that fails without throwing an exception leaves nothing to grep. Nvidia VP of Product Adel el Hallak told The New Stack that an agent can keep running while heading in the wrong direction, carrying an earlier mistake through the rest of a task without producing anything that looks like a conventional software failure ^[1]. He said it sometimes decides to "get creative" when it shouldn't ^[2].

What stands in for a stack trace is a replay of the execution, which el Hallak said can be needed to find where the run went off course ^[3]. "It's not enough to just look at the logs or the inputs and the outputs," he told The New Stack. "It is important to figure out how it got to the answer. What were the reasoning traces? What tools did it utilize? Where did it get stuck? Where did it decide to try a new approach?" ^[4] Answering any of that after the fact requires that something recorded the tool calls and the intermediate steps while the run was live.

Recording costs compute. OpenAI has found that monitoring adds roughly 20% to inference compute for its own systems, according to the same interview ^[5]. On that ratio, five monitored runs bill about what six unmonitored runs do ^[6]. Whether the figure transfers depends on what your monitoring does: exporting spans to object storage costs almost no inference, while a second model reading the first model's output costs another pass over the same tokens.

The 60%-plus failure rate for good coding agents is a measurement of someone else's task mix ^[7]. Fewer than four such tasks in ten complete ^[8]. For that to predict your own fleet, your tasks would have to look like the ones in that sample and your harness would have to look like theirs. Nvidia's NOAH research found that changing the harness while keeping the model fixed can improve agent performance ^[9], so the rate describes the harness and the model together.

The disclosure analogy is where the argument gets thin. A CVE works because it names a package and a version, and the remedy is an upgrade. An agent finding has three candidate owners, because Nvidia splits the stack into model, harness and runtime ^[10], and the account says failures do not necessarily trace back to a single component ^[11]. A report another team can act on therefore has to identify the harness and the runtime, not just the model. "When we find these vulnerabilities, it's not just for one company," el Hallak said. "It's for everyone to patch across." ^[12] The New Stack's article does not identify the companies behind SAFE or say when the exchange starts accepting reports ^[13].

Nvidia's own answer is to make the collection point fixed. El Hallak called OpenShell the one non-negotiable component across the company's reference architectures ^[14]. "You can change whatever harness you need. I'm even open to using whatever models you need," he said. "But the governance, the secure and open runtime that we want to leverage at all times is OpenShell." ^[15] OpenShell is also the layer that handles sandboxing and policy enforcement and provides visibility into an agent's execution ^[16].

El Hallak compared the safety problem to ordinary software testing: "If there's a bug in your software, you don't release it. You work until it's fixed and it passes all your tests." ^[17] That works when a failing test is reproducible. Reproducing an agent failure means reconstructing what happened across the system ^[18], so the trace has to exist before the failure, and capturing it is billed per run.