SageMaker's fallback instance can serve a quantized copy of the same model

A five-entry priority list is a hedge against GPU scarcity, and the hedge changes what the endpoint computes. AWS's example configurations for pool entries are tensor parallelism on high-memory instances, speculative decoding on mid-tier hardware, and quantization on smaller fallbacks ^[10]. So the first choice can serve full weights while the third serves a quantized copy of the same model. The post does not say whether a response identifies which entry produced it.

That matters because SageMaker sells inference by the instance, not by the token ^[6]. A capacity shortage is your outage. Before pools, an endpoint pinned to one instance type failed before serving a single request when that type was unavailable ^[9].

The scale-in rule is the part I would copy. Fallback instances are removed first, so the fleet drifts back toward preferred hardware as capacity frees up, instead of leaving the fallback type in place until the next deployment ^[8]. Running the mixed fleet costs policy work: weighted scaling depends on per-instance-type CloudWatch metric dimensions ^[11]. Pools are supported for single-model, inference component and async endpoints in all commercial AWS Regions ^[12].

The 2x figure is a claim about one model and one goal, throughput optimization on GPT-OSS-20B, at twice the tokens per second for the same request latency ^[16]. Three things would have to hold before that number describes your service. Your goal has to be throughput, because throughput is the goal that selects EAGLE 3.0 speculative decoding in the optimize step, where a latency goal gets kernel tuning instead ^[14]. Your traffic has to resemble what NVIDIA AIPerf generated on the benchmark run ^[14]. And your baseline has to be the configuration AWS improved on, not a serving stack you have already tuned by hand. The output is a Model Package carrying time to first token, inter-token latency, P50/P90/P99, throughput and a cost projection ^[15].

AWS says the manual version of that selection is two to three weeks of benchmarking against more than 1,000 combinations ^[13]. In practice teams benchmark four combinations and ship the one that fits. Generating the recommendation costs nothing, and customers with ML Reservations can run the benchmarks on reserved capacity at no extra charge ^[17].

On the fork itself, AWS describes managed endpoints for teams that want AWS to handle infrastructure and operations, and HyperPod Inference for teams that need Kubernetes-native control over dedicated GPU clusters ^[2], with a table comparing the two paths across seven dimensions ^[19]. Seven of the thirteen launches are on the endpoint path ^[3], which leaves six on the HyperPod side ^[4]. AWS's own statement of what makes this hard names cold starts that span multiple minutes as containers and weights transfer, constrained GPU capacity, and monitoring that exposes none of the token-level signals that matter in production ^[5]. Observability is listed among the areas the seven endpoint launches cover ^[3]; the three detailed in the post are instance selection, capacity fallback, and an /openai/v1 path serving Chat Completions with streaming ^[13][7][18].