Build1 publisher2 min readPublished
A CreateAIBenchmarkJob call ramps a SageMaker endpoint from 64 to 1,024 concurrent requests
Amazon's concurrency sweeps push rising traffic at a SageMaker inference endpoint and report where throughput stops improving. Three vLLM settings in the sample deployment decide whether that curve transfers to your traffic.
The Engineer · Build desk
What happened
- Concurrency sweeps are built into Amazon SageMaker AI Inference Recommendations, which AWS says removes the need to build or maintain custom load-testing infrastructure.
- The documented workflow has four steps: deploy on the native vLLM container, configure a workload profile with input and output token counts and streaming mode, call the CreateAIBenchmarkJob API, then analyze the results.
- The sweep raises concurrency in stages, for example 64 to 256 to 1,024 simultaneous requests, to trace the point where more traffic stops improving throughput and starts degrading latency.
- It returns three planning numbers: the concurrency with maximum throughput at acceptable latency, the level where latency crosses the SLA, and the instance count needed to cover peak traffic.
- The example deploys NVIDIA Nemotron-3 Nano 30B, a Mixture-of-Experts model with 3B active parameters, to an ml.g7e.2xlarge instance backed by an NVIDIA Blackwell GPU.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost At the ratio AWS uses to open the post, five instances doing one instance of work, four fifths of the GPU spend buys nothing, and the bill lands on whoever signed the capacity request.
- constraint A single pass over rungs that quadruple each time cannot resolve the saturation point better than a factor of four, so a team either pays for extra runs or provisions to the safe rung.
- decision Anyone reusing a published instance count now has to reproduce the serving configuration it came from, including prefix caching and the 0.85 memory setting, or run the sweep against their own container.
- capability Because the sweep is an API job, a capacity number can be regenerated after each model or container upgrade instead of being re-established by hand each time.
Three SM_VLLM_* environment variables in the sample deployment decide what the sweep actually measures [11]. SM_VLLM_ENFORCE_EAGER is set because Nemotron-3 Nano uses a Mamba-Transformer hybrid architecture that requires eager execution mode [12]. GPU memory utilization is 0.85; AWS says that leaves headroom for KV cache growth under high concurrency [13]. Prefix caching is on so repeated system prompts reuse cached key-value pairs [14].
The benchmark measures the endpoint as configured, so the curve belongs to those three settings. For the throughput figure to mean anything for your fleet, your traffic has to repeat system prompts at something like the rate the workload profile assumes; the profile sets input and output token counts and streaming mode [8]. A workload of mostly unique prompts gets fewer cache hits than the benchmarked one did. In my view the useful output is your own curve against your own container and prompt mix, and the published configuration is where you start.
The illustrated ladder steps 64, then 256, then 1,024 simultaneous requests [6]. Both 256/64 and 1,024/256 are 4, so a single pass over those rungs locates the saturation point inside a factor of four [15]. A knee at 300 concurrent requests looks like a healthy run at 256 and a failed one at 1,024. Narrowing it takes runs at intermediate levels, and each run is another CreateAIBenchmarkJob call [8].
The alternative AWS describes is deploying, load-testing manually, adjusting, and repeating until the numbers look acceptable [2]. "Choose five ml.g7e.2xlarge instances when one would suffice, and you burn your budget on idle GPUs," the post says [3]. That five-instance figure sits in the post's framing of the problem, not in results from the Nemotron-3 Nano run.
Adoption cost is the prerequisite list: an AWS account with SageMaker AI access, an IAM execution role with permissions for SageMaker AI and Amazon S3, and service quota for ml.g7e.2xlarge endpoints [9]. Quota binds the comparison, because a sweep can only rank instance types you are allowed to launch. The post text available here breaks off during step two, before any measured throughput or latency number appears [17].
What to watch
- Whether the sweeps extend beyond the native vLLM container and ml.g7e.2xlarge to other serving stacks and instance families.
- Whether AWS publishes the measured tokens-per-second and latency figures from the Nemotron-3 Nano run, and the prefix-cache hit rate behind them.
- Whether CreateAIBenchmarkJob accepts caller-specified concurrency levels between the sample rungs.