Build1 publisher2 min readPublished
HyperPod holds the inference deployment until every target node has cached the weights
Reading a 600 GB checkpoint from local NVMe at 7 GB/s takes about 86 seconds, so the network stops gating scale-out on warm nodes. The first fill still costs the full download. The slowest node sets it.
The Engineer · Build desk
What happened
- AWS has launched model caching for Amazon SageMaker Inference on HyperPod, which pre-loads model weights and container images onto cluster nodes before any pod is scheduled to need them.
- Without it, a starting pod waits on two sequential downloads, first the inference server image from ECR and then the model weights from S3, FSx for Lustre or HuggingFace Hub.
- AWS puts the image pull at 5 to 7 minutes, the weights for a 145 GB model on S3 at 20 minutes or more, and a 600+ GB model like DeepSeek-R1 at upwards of 30 minutes.
- The cache survives pod restarts on the same node, and during scale-out a new pod landing on a node that already holds the weights starts immediately at roughly 7 GB/s of local read.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability On nodes that are already cache-ready, the figure an autoscaling policy has to budget for is a local read of about 86 seconds for a 600 GB model. Target utilisation and cooldown windows can be tuned instead of padded for a half-hour warmup.
- constraint The documented immediate start applies to pods that land on nodes which already hold the weights, so the benefit depends on keeping enough cached nodes in the pool to absorb a spike.
- cost Every target node stores its own full copy of the weights. A 600+ GB model claims 600+ GB of local NVMe on each node in the group, and that capacity stays committed to the cache.
- decision Teams now choose between a weights cache that withholds the deployment until the slowest node finishes and an image cache that ships the deployment straight away without waiting for the pull.
Divide the checkpoint size by the read speed and the claim gets specific. A 600 GB model at 7 GB/s is about 86 seconds of local read [1]. A 145 GB one is about 21 seconds [2]. AWS says pods can typically start serving traffic in seconds rather than tens of minutes once caching is on [7]. For DeepSeek-R1 that means roughly a minute and a half, assuming the loader can saturate the NVMe device while it deserializes weights onto the GPU.
The same division on the download figures gives the throughput the old path was living with. 145 GB in 20 minutes works out to about 121 MB/s [3]. 600 GB in 30 minutes is about 333 MB/s [4]. AWS attaches the caveat itself, putting the 145 GB case at "20+ minutes depending on network conditions and available bandwidth" [4]. So the roughly 58x gap between 7 GB/s and 121 MB/s [5] describes the network path in front of one pod, not a ceiling on the storage service. If your nodes give each pod more bandwidth than that, your baseline is faster and your speedup is smaller.
AWS frames the cost as an autoscaling gap. When a HorizontalPodAutoscaler asks for five new pods, all five run the download sequence independently. The policy reacts in seconds while the extra traffic waits 25 to 30 minutes or more [14]. The HyperPod Inference Operator creates a ModelDataCacheConfig, downloads the weights to local NVMe on all target nodes, and labels each node cache-ready [9]. Only then does it create the inference deployment, so that pods can always access local data [10]. Time to first deployment is therefore set by the slowest node's download, and the deployment waits on the whole group. That barrier is what makes the local-read guarantee unconditional.
Image caching is deliberately looser. The operator creates a DaemonSet that pulls the container image onto all target nodes, and it creates the deployment immediately, because the image cache does not block deployment creation [12]. A pod that starts with the image already present skips the ECR pull and saves 5 to 7 minutes [13]. Nothing in that documented sequence guarantees the DaemonSet finished before the first pod is scheduled.
The post says it shares benchmark results [15]. No figures appear in the text available here [16]. For the numbers that are here to transfer to your cluster, the NVMe read has to be the binding constraint on startup. Deserialization, tensor-parallel sharding and CUDA graph capture have to sit behind it. And every target node holds its own copy: caching a 600+ GB model spends 600+ GB of local NVMe per node [6].
What to watch
- Whether the published benchmark measures pod-ready time or time to first token, and which storage backend the uncached baseline pulled from.
- Whether the operator gains a partial-readiness mode so a deployment does not wait on the slowest node's download.
- How cache refresh and eviction are defined when a model version changes and nodes still hold the old weights on NVMe.