Build1 publisher3 min readPublished
NVFP4 and cache reuse cut MLPerf's edge agent workload to 24 minutes on one Jetson Thor
NVIDIA's submission runs the same Qwen3.6-27B as the llama.cpp reference on the same Jetson board and finishes 6.4x sooner. Most of the gap comes from prompt tokens the runtime never has to prefill.
The Engineer · Build desk
What happened
- NVIDIA's TensorRT Edge-LLM submission finished all 1,007 turns of the MLPerf Inference v6.1 Edge Agentic performance workload in 24 minutes 36 seconds at 52.33 tokens per second, 6.4x faster than the llama.cpp reference.
- The MLCommons reference run uses the same Qwen3.6-27B model on the same Jetson AGX Thor board, quantized to Q4_K_M under llama.cpp, and takes 2 hours 37 minutes.
- NVIDIA attributes the result to three techniques used together: NVFP4 quantization, tree-based multi-token prediction, and KV cache reuse.
- The performance phase replays 20 recorded software-engineering agent trajectories against an OpenAI-compatible endpoint, with input length growing to about 23.5K tokens late in a conversation.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision For a team already running Thor hardware, the upgrade decision is which runtime and which checkpoint, because the same 27B weights sit under both the 24-minute and the 2h37m number.
- constraint A large share of the speedup rides on a 96 percent prompt-cache hit rate, so an endpoint that interleaves users or rebuilds its system prompt each turn cannot inherit that part of the result.
- exposure MLPerf's accuracy phase is single-turn with reasoning off, so the risk that 4-bit activations degrade tool-call arguments deep into a conversation falls on the deployment team's own evaluations.
- cost The porting cost is one post-training quantization pass on a development system, or none if the published calibrated NVFP4 checkpoint is used, and the memory the smaller weights free up is available for context and speculative-decoding state.
Low-batch decoding on an edge board is heavily bounded by DRAM bandwidth, according to NVIDIA's write-up, so the bytes moved per token set the decode ceiling [9]. That makes the number format the first thing to change. The submission puts weights and activations in NVFP4, including the language-model head, and holds the KV cache in FP8 [10]. NVFP4 is a 4-bit floating-point format the Blackwell GPU in Jetson AGX Thor supports [10].
The two remaining techniques depend on the shape of the conversation, not on the board. Tree-based multi-token prediction cuts the number of target-model steps during generation, and prefix reuse cuts the prefill before generation starts [16]. Qwen3.6 is a hybrid architecture, so the runtime restores recurrent state and partial KV-page state alongside the cached attention pages, then prefills only the new suffix [14]. On this trajectory set that came to about 96 percent of prompt tokens served hot, with roughly 0.5M of 13.6M prompt tokens actually prefilled [15]. That leaves 13.1M prompt tokens the machine never processed [5].
Per turn the work is small. 1,007 turns in 1,476 seconds is 1.47 seconds a turn [1]; the reference run's 9,420 seconds works out to 9.4 [2]. The average prompt is about 13,500 tokens [4], and input reaches roughly 23.5K tokens late in a conversation [5]. If 52.33 tokens per second is the average across the whole wall clock, the run emitted about 77,000 tokens, near 77 a turn [6].
Whether that gap transfers depends on conditions the hardware does not guarantee. The endpoint serves one conversation at a time, since the submission ran SingleStream [7]. Unified memory on the board comes to 128 GB, and the capacity freed by 4-bit weights goes to long context, speculative-decoding state and the application itself [7][11]. Prompts have to be append-only, so a stale prefix never invalidates the cache. Put a clock in the system prompt, or round-robin two users through one endpoint, and you slide toward the cold case, where the model prefills the shared history again on every turn and the cost grows with the conversation [13]. NVIDIA's post opens by framing edge inference as work inside a limited power and memory envelope [17], and the run was made at the MAXN power mode [7].
MLCommons publishes the llama.cpp run as its Edge Agentic example, using the same Qwen3.6-27B at Q4_K_M [8]. Q4_K_M against NVFP4 is a format difference on top of a runtime difference, and NVIDIA credits the result to quantization, tree-based MTP and KV cache reuse together [3].
The accuracy gate is narrower than the workload it accompanies. In the performance phase, an Intersection over Union inline check confirms the agent is running correctly [5], while the accuracy phase uses Berkeley Function Calling Leaderboard v4 prompts, single-turn only and reasoning off, which the post says balances accuracy against evaluation time on edge devices [6]. Nothing in that score tests whether NVFP4 activations still produce valid arguments 40 turns into one conversation.
For a team already running llama.cpp on Thor, quantization is cheap: start from the published calibrated Qwen3.6-27B NVFP4 checkpoint, or do post-training quantization once on any development system before deploying [12]. Reuse depends on whether their agent appends to one conversation or rebuilds its prompt each turn.
What to watch
- A tuned llama.cpp submission on the same Thor board would separate the runtime gain from the NVFP4 versus Q4_K_M format gain.
- Whether a later Edge Agentic round scores accuracy on multi-turn trajectories instead of single-turn BFCL v4 prompts with reasoning off.
- Whether other teams reproduce the decode rate from the published calibrated NVFP4 checkpoint on lower-memory or lower-power Thor configurations.