Build1 publisher2 min readPublished
NVIDIA's AIPerf drops the Perf Analyzer stack for worker processes over ZMQ
The designated successor to GenAI-Perf splits load generation from record processing so the benchmark client stops hitting Python's GIL. Teams holding a GenAI-Perf baseline inherit a port and a re-run.
The Engineer · Build desk
What happened
- NVIDIA published a walkthrough for AIPerf, which it calls the designated successor to GenAI-Perf and a ground-up rewrite rather than an increment on the existing tool.
- AIPerf runs as a multiprocessed system, with worker processes generating load, separate record-processor services handling results, and coordination over ZMQ.
- The tool covers 15-plus endpoint types, including chat, responses, NIM rankings and image generation, alongside ShareGPT and trace replay from Mooncake, Baseten and WEKA's AgentX.
- Load shape is configurable: constant, Poisson and gamma arrivals with tunable burstiness, gradual ramping of concurrency or request rate, and range-ratio distributions for variable input and output lengths.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision A team holding a GenAI-Perf baseline taken at high concurrency has to decide whether that number still stands. NVIDIA's own account puts the old client's single-process ceiling inside the measurement path.
- constraint Any AIPerf throughput figure quoted without min_tokens and ignore_eos set is not comparable to a second run, so reviewers need the command line.
- cost ARM benchmark images need a C toolchain added before AIPerf will install, and that work lands on whoever maintains the aarch64 CI image.
- capability Replaying captured production traffic and shaping burstiness now sit in the same harness as the synthetic smoke test. A team can stop maintaining two load generators with two sets of definitions.
Four flags in NVIDIA's example command decide what the run actually measures. Setting `--synthetic-input-tokens-stddev 0` and `--output-tokens-stddev 0` pins every request to exactly 128 input and 128 output tokens. The post says that reproduces a commonly used static benchmark that holds request and output lengths constant [15]. Every request in that run carries the same 256 tokens [19]. A 128/128 run at a fixed arrival rate predicts your deployment only if your prompt and completion lengths cluster near those values and your traffic arrives that way.
The other two flags decide whether the server does what you asked. `--extra-inputs min_tokens:128` and `--extra-inputs ignore_eos:true` make the model emit the full 128 tokens instead of stopping when it naturally finishes. Without them, NVIDIA says, throughput comes out lower than it should be and is not reproducible across runs [17]. The post puts it this way: without those flags, "the output token count is a suggestion" [16]. `--streaming` is not optional either, because without it the server batches the full response and TTFT and ITL cannot be measured [18].
Porting is more than translating flags. GenAI-Perf ran on top of Perf Analyzer and AIPerf does not, a break NVIDIA calls the reason AIPerf can scale [2]. NVIDIA also says most benchmarkers, GenAI-Perf among them, use a single process that becomes GIL-bound under real concurrency or request rate [4]. AIPerf's worker processes and separate record-processor services keep the client from becoming the bottleneck, it says [6]. If your current TTFT or throughput baseline came from a high-concurrency run under the old client, part of that number may describe the client. The post links a migration guide for the key deltas [3]. It does not state a deprecation date for GenAI-Perf [20].
NVIDIA's list of what AIPerf is meant to replace is curl commands, a hand-rolled asyncio script, and one more one-off load generator, and the objection to all of them is a single process that caps concurrency at the GIL and numbers measured against a reference you built yourself [10]. The reference you built yourself is the harder problem.
Install is one uv command, either as a tool or into a virtual environment [13]. On aarch64 the `crick` dependency ships source-only and needs a C toolchain, build-essential on Debian and Ubuntu or Development Tools on RHEL [14]. The walkthrough serves Qwen3-0.6B through vLLM with the qwen3 reasoning parser on port 8000 [12], picked because it fits one GPU and iterates fast. NVIDIA says swapping in another model or endpoint afterwards is a one-flag change [11].
What to watch
- Whether NVIDIA publishes a deprecation schedule or end-of-life date for GenAI-Perf and Perf Analyzer.
- Whether crick starts shipping aarch64 wheels, which would remove the C toolchain step from ARM benchmark images.
- Whether the migration guide's deltas include changed metric definitions that would invalidate stored TTFT and ITL baselines.