Build1 publisher3 min readPublished
A second scaling policy set this ASG flapping between five and six instances
An operator writing on dev.to describes a CPU target-tracking policy and a per-instance free-slot step policy taking turns on the same Auto Scaling Group, with tripling the alarm window only slowing the oscillation to six minutes.
The Engineer · Build desk
What happened
- A fleet of stateful worker instances ran behind an Auto Scaling Group with an On-Demand floor for baseline capacity and Spot instances above it for burst, each instance holding a bounded pool of session slots.
- Scaling started with target tracking on fleet-average CPU, which could not see one instance sitting at zero free slots and rejecting sessions while three idle instances held the average down.
- A CloudWatch alarm on per-instance free-slot count was wired to a step scaling policy, and the group then flipped between five and six instances every 30 to 90 seconds for minutes at a time.
- Both policies were removed in favour of one target-tracking policy on a Metric Math occupancy expression targeting 75 percent, after which describe-policies returned a single policy and the oscillation stopped.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Anyone running target tracking beside step scaling on one group has to pick which policy owns the group, because in this account the two acted on their own metrics and neither could override the other.
- constraint The SEARCH() restriction in metric alarms forecloses the obvious design: a dynamic fleet-wide aggregate without hardcoded instance IDs cannot be expressed in the alarm that a step policy depends on.
- cost Tuning the window adds detection latency: a ten-minute breach window means a saturated instance waits up to ten minutes for relief, and the fleet kept oscillating anyway.
- exposure Because the replacement expression averages free slots across the fleet, the 25 percent headroom above target is what covers a single saturated host with idle neighbours. Nothing in the policy sees that host directly.
The two policies were reading different systems. Target tracking on average CPU is a claim about the fleet. An alarm on free slots for any single instance is a claim about one host. After a burst the average fell, the CPU policy scaled in, the resulting per-instance squeeze tripped the free-slot alarm, and the step policy scaled back out, according to the post [6]. Neither policy knew the other existed, and neither was wrong on its own metric [6].
The activity log showed the group moving between five and six instances every 30 to 90 seconds, for minutes at a stretch [7]. At that period the group is resizing roughly 40 to 120 times an hour [1].
The first attempt was timing. The author lengthened the pool alarm's sustained-breach window from three minutes to ten and redeployed [8]. The oscillation continued on a five to six minute cadence [9], which works out to 10 or 12 resizes an hour, a factor of four to ten off the original rate [1][2].
The platform detail that shaped the rewrite is a restriction on alarms. `SEARCH()` is not accepted inside a CloudWatch metric alarm; `PutMetricAlarm` rejected the author's first fleet-wide aggregate outright, and the post says that limit was confirmed against AWS documentation [10]. Target tracking policies have supported Metric Math since December 2022, and the post attributes the difference to target tracking evaluating the metric expression directly instead of routing it through the alarms evaluation engine [11]. So a dynamic fleet aggregate can be built in the scaling policy; the alarm a step policy requires will not take it [10][11].
The replacement is one target-tracking policy on `occupied_per_instance = pool_max_size - avg(free_slots)`, aimed at a 75 percent utilisation band [12]. `terraform plan` showed one resource added and three removed with nothing else touched, `describe-policies` afterwards returned exactly one policy, and the author reports no oscillation since [13].
`avg(free_slots)` is a fleet-wide number, the same class of signal as average CPU, which is the property that made CPU miss a saturated host in the first place [3]. A 75 percent target leaves headroom, so the group scales out earlier than the CPU policy did. One instance at zero slots beside three idle ones still does not move the average into the scale-out band by itself. Where sessions are pinned and cannot be placed on an instance with room, headroom is what keeps them placeable and the per-instance signal is no longer in the loop. The post does not report session rejection rates after the change.
The author's own framing is that this is not an AWS bug: it is "a distributed-systems pattern that shows up anywhere two control loops react to different observations of the same underlying system state, without a shared source of truth" [14]. I think that reading is right, and it is also one operator's account of one fleet, with the numbers taken from their own scaling activity log [7]. For the conclusion to transfer, two policies have to be attached to the same group with metrics that can disagree about direction of travel, and the constraint that actually hurts has to live on an individual instance where a fleet aggregate can hide it.
What to watch
- Whether session rejections reappear once a single instance saturates while fleet occupancy sits under the 75 percent target.
- Whether AWS ever accepts SEARCH() inside metric alarms, which would reopen the step-scaling path for dynamic fleet-wide aggregates.
- How the occupancy expression holds up when pool_max_size stops being a single constant across instance types.