Build1 publisher3 min readPublished
Grading a trace twice separates where an agent broke from whether the task finished
NVIDIA's developer blog sets out the five-level rollup from step to benchmark and the two scores that read the same trace. Step-level says where the chain broke. End-to-end reads the environment and says whether the refund posted.
The Engineer · Build desk
What happened
- NVIDIA's developer blog traces agent evaluation from scoring a single function call to scoring whether a whole task finished, and says nearly every serious agent benchmark now rests on tool use.
- The Berkeley Function-Calling Leaderboard grades individual calls, so a valid issue_refund call can pass on call accuracy while the task fails because underlying checks or updates were skipped.
- Every run rolls up through a fixed hierarchy of benchmark, trial, task, turn, and step, where a step is one atomic action inside a turn such as a tool invocation, a plan, or the final message.
- Step count is often the axis that varies most between models on the same task, four steps against fifteen, though on suites like Terminal-Bench 2.0 steps-per-turn varies as well.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision A team that gates its release on the end-to-end score still has to decide what, if anything, a step-level score blocks, because the same trace now feeds a shipping gate and a debugging queue.
- constraint A pass rate measured on single-call, single-turn tasks cannot bound the risk on a fifteen-step stateful deployment, so the published number does not transfer unless the harness matches the task complexity.
- cost Buying step-count reductions through parallel tool calling pays out in latency and leaves the per-call spend where it was, so efficiency claims quoted in steps overstate what comes off the bill.
- exposure Shipping on one trial's success rate leaves variance unpriced, and the model that swings between 90% and 74% looks like the stronger pick until the bad run lands on customer traffic.
A trace is the ordered log of one attempt: the user message, each step, and the environment state when the attempt stops [10]. Process scoring grades the rows. End-to-end scoring grades the final state [10]. Both readings come off the same object, so a harness that already stores traces can add outcome scoring without collecting anything new.
Step-level scoring asks whether a call was valid, relevant, and useful given the state at that point [7]. End-to-end ignores the path and asks whether the refund posted and whether the ticket routed correctly [7]. The second score collapses a failure on step one and a failure on step nine into the same "task failed" [8]. NVIDIA's post says most production evals gate the release on end-to-end and keep step-level tracing underneath for debugging [9].
Under both sits a tool-calling benchmark that scores three things in order: deciding to use a tool, selecting the right one, and populating its arguments [11]. According to the post, a model that reaches for a tool when a direct answer would do fails as surely as one that skips a tool it needed [12]. Cost and latency ride on top, set by the call's verbosity and runtime [13].
Parallel tool calling cuts step count and latency and leaves call count alone: a one-step turn firing four tools still issued four calls [17]. Because the spend is set per call, the parallel version pays for the same four calls it would have paid for serially [24]. The post's rule for rolling up is to go in order and not average steps into a benchmark score [18]. A mean step count is a number that survives exactly one question at review.
Success rate is paired with consistency for a reason: a model that hits 90% on one trial and 74% on the next is a worse bet than one holding 84% [19]. Those two trials average 82%, two points under the steady model [23]. A trial is one independent pass over the whole task set under a fixed configuration [14], so that spread is not a configuration difference. Pairing tool-call precision with argument accuracy catches the other gap, which is slot filling [19].
Two benchmarks can both claim to test tool calling and produce numbers that are not comparable, and NVIDIA names task complexity, statefulness, and methodology as the three dimensions behind most of the gap [20]. For a published pass rate to say anything about a deployment, the harness has to match on the first two. A single-call benchmark will not tell you whether a model collapses on step eight of fifteen [21]. A static environment will not surface the drift, context loss, and corrupted state that appear once the environment updates on every action [22]. The post does not quantify how many teams are still scoring single calls [25].
The arc it describes is short. The original harnesses were built for static tasks, and the first model-agnostic open-source harness decoupled the model from the evaluation protocol [3]. The Berkeley Function-Calling Leaderboard then scored function selection and argument accuracy across single- and multi-turn scenarios [4]. Full agentic evaluation now wants an execution environment that runs each call, tracks state, and reads the world afterwards [6].
What to watch
- Whether tool-calling leaderboards begin publishing consistency across trials next to a single success rate.
- Whether BFCL-style harnesses start reading final environment state at the end of a run instead of grading calls alone.
- Whether suites like Terminal-Bench 2.0 report steps-per-turn distributions alongside pass rates.