Skip to content
Today
Sign in

BuildNot yet confirmed elsewhere1 publisher2 min readPublished

Amazon ECS now drains and cycles instances whose GPUs report hardware faults

Amazon ECS now reads GPU health from NVIDIA's DCGM and drains and cycles faulty instances on its own, the ECS team wrote in The New Stack. For GPU fleets on Managed Instances, the detect-drain-replace loop that operators used to script is now AWS's job.

The Engineer · Build desk

How we use AISend a correction

Illustration accompanying Amazon ECS now drains and cycles instances whose GPUs report hardware faults
Generated illustration

What happened

  • The trigger the ECS team describes is a GPU whose correctable ECC errors escalate into faults, slowing down or failing the tasks on that instance.
  • Instances impaired by a network partition, a thermal event or EBS volume degradation are also handled, since their tasks would otherwise keep running with no orchestration behind them.
  • On ECS Managed Instances, ECS also patches the OS, kernel and GPU drivers itself, rolling updates out gradually and rolling back when it detects a problem.
  • On AWS Fargate, the post says, a bad instance, a bad OS update or a driver regression is AWS's to detect and recover from.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • decision On Managed Instances, pages that fire on GPU hardware faults now report work ECS is already doing, so teams have to decide whether those alerts should still wake a human.
  • constraint Teams inherit ECS's choice of which GPU error classes count as a hardware fault. A runbook tuned to pull hosts earlier no longer sets that threshold.
  • exposure The post ties GPU health monitoring to Managed Instances, so teams running GPU hosts on capacity they manage themselves should keep their own detect-drain-replace loop until AWS says otherwise.

The case for handing this off rests on time in service. According to the post, a degraded instance puts all of its scheduled tasks at risk, and the failure spreads further for as long as the instance remains in rotation [6]. Teams that wanted bad hosts out fast have had to build the loop themselves [3]:

1. Detect the impaired instance. 2. Drain its workloads. 3. Cycle it.

After that comes the job with no end date: keeping that automation reliable at scale [3]. "ECS now does this for you automatically," the post says [3]. The ECS team wrote the post in the first person, so this is AWS describing its own service [7].

Detection is the step I would read twice. ECS integrates with NVIDIA's Data Center GPU Manager and, in the post's words, "watches for error classes that indicate a genuine hardware fault" [4]. Filtering on error class is the right design. It lets ECS tell a GPU that logs errors apart from one that has actually failed. The part of the post on record does not list those classes or say what thresholds trigger a drain [4].

The team's position is that "you shouldn't need to build your own detection loops and remediation runbooks for failures ECS can handle itself" [13]. Keeping an old loop running beside it means two automations draining the same host, which is a slow way to find out which one is faster. The post describes the feature as AWS taking patterns it uses to keep its own systems healthy and offering them to customers, "often as sensible built-in defaults" [12].

The shared responsibility split still applies. AWS owns resilience of the cloud. The customer owns resilience in the cloud, meaning how the application withstands and recovers from failure [11]. For ambiguous cases, where the right response depends on the workload, ECS exposes the same controls so the customer can make the call [10]. In my view, those are the runbooks worth keeping. Whether to drain a GPU that is degraded but still serving is a workload decision [10]. How the application recovers when ECS pulls an instance out from under it is the customer's half of the model [11].

What to watch

  • Whether ECS documentation publishes the DCGM error classes and thresholds that trigger a drain, so teams can check them against their own alert rules.
  • Whether AWS extends DCGM-based GPU health detection to ECS capacity that customers run themselves on EC2.
  • What event ECS emits when it cycles a GPU instance, and whether teams can alert on it without acting on the host.

Clarity's read

What the record supports and how the coverage leans. The claims behind it follow.

Reality

Evidence35
Adoption
Insufficient
Hype gap+20
Incentives80
Confidence40
Why these scores

Claim ledger

Ranked by verification strength, evidence, and original report placement.

  1. [1]

    If a GPU on an accelerated instance in an inference fleet starts throwing correctable ECC errors that escalate into faults, the tasks on that instance slow down or fail.

    ReportedSupportedSource: Amazon ECS team, via The New Stack2 sources— create a free account to open themView cited source
  2. [2]

    If an instance becomes impaired because of a network partition, a thermal event or Amazon EBS volume degradation, its tasks keep running in place with no orchestration behind them and eventually become impaired themselves; ECS now handles such impairments automatically.

    ReportedSupportedSource: Amazon ECS team, via The New Stack2 sources— create a free account to open themView cited source
  3. [3]

    Dealing with impaired instances yourself would mean building automation to detect impaired instances, drain their workloads and eventually cycle them, then maintaining and operating that automation reliably at scale. "ECS now does this for you automatically."

    ReportedSupportedSource: Amazon ECS team, via The New Stack2 sources— create a free account to open themView cited source

Sources

1 independent publisher whose own reporting we read for this story.

  1. thenewstack.io

    1 article · October 9, 2026

    Amazon ECS now auto-repairs failing GPUs and instances. Here’s why it matters for SREs.

Share your take

Let Clarity write the post for you.

Signed-in readers get a short post drafted on this story in the register they choose — narrative, analytical, or a direct position — editable to the last word before it goes anywhere. The share buttons at the top of this story work without an account.

Create a free account to composeSign in

Topics and entities

Follow any of these and your For You feed starts watching them — no settings page required.

Topics

Entities

Loading related stories