Product 1 publisher3 min readPublished September 16, 2026 at 3:06 AM PDT

How one shared policy library ran a provisioning service out of memory

A policy library that behaved well when an engineer ran it on one storage instance at a time exhausted a provisioning service's memory once the same calls ran in parallel. Nothing in it was broken.

The Product Desk · Product desk

Illustration accompanying How one shared policy library ran a provisioning service out of memory

What happened

A large-scale provisioning system created storage partitions on demand and applied an access-control policy to each new one through a shared library whose other caller was an operator CLI.
The provisioning service applied those policies concurrently, so one request could fan out into many simultaneous calls to the library. On ordinary requests that improved throughput.
When a much larger request arrived, each parallel operation loaded the state it needed independently and the combined memory footprint grew until the service exhausted its memory limit.
The account's proposed fix is to make resource constraints explicit and use signals such as memory pressure, concurrency and queue depth to control how much work the system accepts.

Compiled by The Product DeskSomething wrong?How this is made

Why it matters

constraint Reusing a shared component takes a number that sits outside its interface: bytes per call, measured under the new caller's concurrency. Without it, the second caller cannot divide its memory headroom into calls.
decision Admission control becomes a sizing decision the team has to make on purpose. Cap request size against a worst-case unit cost, or throttle on live memory pressure. The two put the refusal in different places.
exposure The requester who submits the largest valid request is the one who takes the service down, having done nothing wrong and unable to see from outside that the job was oversized for the budget.
precedent A library with two callers of different shapes is two products with two budgets, and a serial benchmark that used to be sufficient evidence becomes evidence about only one of them.

Picture the original caller: one command, one storage instance, the current policy state read into memory, the changes computed and applied ^[2]. The footprint is bounded, and the caller is what bounds it. The number could stay implicit, because a person at a terminal cannot run fifty of those at once.

The provisioning service reused the function and left that assumption behind in the CLI ^[3]^[6]. Per-call memory cost stayed the same, and the multiplier went up ^[1]. The same code path was safe at one call and over the limit at high fan-out, and the interface looked identical either way.

Then the timing. The library had been optimized shortly before the incident, and the tests ran mainly against the original serial pattern ^[7]. A serial benchmark measures the cost of one call, and the outage came from the sum of many concurrent ones ^[2].

A second caller adopts the documented interface and also a cost model the interface omits: bytes per call, and how many calls at once are survivable. In this system the size of a provisioning request went unchecked against its expected resource cost, and concurrent work ran untied to the memory pressure those policy operations created ^[8]. Every part behaved sensibly, including the requester, who submitted a valid request ^[5].

devops.com put the needed rule as "The provisioning service must not accept or execute more work than it can process within its resource budget" ^[9]. The publication also concedes the awkward part: a fixed concurrency limit helps, but the safe level moves with request size, state size, traffic and available memory ^[10]. So the budget has to be an input to the decision, fed by signals such as memory pressure, concurrency and queue depth ^[11]. In my view that trade is worth taking with eyes open, because on the requester's side the improvement arrives as a job that gets queued or refused for reasons they cannot see from outside.

Two questions sort this for any shared component about to acquire a second caller. Has the per-unit resource cost been measured under the new caller's concurrency? And can a single request contain an unbounded number of units?

Measured cost with a capped request size is the safe case. Measured cost with an uncapped request gives you a divisor: headroom divided by worst-case unit cost is the largest request you can admit. Unmeasured cost with a cap means the cap is a guess that has held so far. Worth knowing before someone raises it for throughput. Unmeasured cost with no cap is the provisioning service, working fine until a request arrives that is large enough to find the memory limit ^[4].

The account was published without the service's memory limit, the concurrency level, or the size of the request that exhausted it ^[3]. Anyone applying the shape to their own system will have to measure those numbers there.

What to watch

Whether the follow-up publishes the specific counters and thresholds the provisioning service ended up admitting work against.
Whether owners of shared internal libraries start publishing a per-call resource cost alongside the API contract.
Whether the next incident of this shape involves a caller other than a provisioning service. That would tell you the pattern is about fan-out.

Claim ledger

Ranked by verification strength, evidence, and original report placement.

[1]
A large-scale provisioning system created storage partitions on demand and, as part of that process, applied an access-control policy to every new partition.
ReportedSupportedView cited source
[2]
The policy logic lived in a shared library whose other user was an operator CLI that engineers used to apply policies manually, one storage instance at a time; each invocation read the current policy state of one instance into memory, calculated the required changes and applied them, keeping the work and memory footprint reasonably bounded.
ReportedSupportedView cited source
[3]
The automated provisioning service later reused the same library with a different usage pattern: it could create many partitions at once and apply policies concurrently, so a single request could fan out into many simultaneous calls to the policy library, which worked for ordinary requests and improved throughput.
ReportedSupportedView cited source
[4]
When a much larger request arrived, the provisioning service accepted it and started processing many policy operations in parallel; each operation independently loaded the state it needed from the storage service, and the combined memory footprint kept growing until the provisioning service exhausted its memory limit.
ReportedSupportedView cited source
[5]
According to the account, every component was doing something reasonable: the policy library worked as designed, the provisioning service used concurrency to improve throughput, and the requester submitted a valid request, so the failure came from how those behaviours combined.
ReportedSupportedView cited source
[6]
The policy library had an implicit assumption built into its original design, that its resource cost was safe because callers would use it in a bounded way; that assumption was true for the CLI but no longer true once the same operation was used concurrently by the provisioning service.
ReportedSupportedView cited source
[7]
The library had been optimized shortly before the incident, but those improvements had been tested mainly against the original serial usage pattern, which told the team little about the aggregate memory cost of many concurrent calls.
ReportedSupportedView cited source
[8]
The original assumption had never been turned into an explicit system constraint: nothing limited the size of a provisioning request based on its expected resource cost, and nothing tied the amount of concurrent work to the memory pressure created by those policy operations.
ReportedSupportedView cited source
[9]
"The provisioning service must not accept or execute more work than it can process within its resource budget."
ReportedSupportedSource: devops.com, stating the system-level rule the design neededView cited source
[10]
A fixed concurrency limit can help, but the safe level may change with request size, state size, traffic or available memory; the important part is that the provisioning service can account for resource pressure when deciding how much new work to accept.
ReportedSupportedView cited source
[11]
The deeper fix is to make resource constraints explicit and use signals such as memory pressure, concurrency and queue depth to control how much work the system accepts.
ReportedSupportedView cited source
[d1]
The per-call memory cost of the policy operation was unchanged between the two callers; what changed was the number of calls running at once, so the same code path was safe at a fan-out of one and exceeded the memory limit at high fan-out.
DerivedSupported1 source— sign in to open
[d2]
A benchmark run against the original serial usage measures the cost of a single call, so it can pass while the sum of concurrent calls exceeds the service's memory budget.
DerivedSupported1 source— sign in to open
[d3]
The published account does not state the provisioning service's memory limit, the concurrency level in use, or the size of the request that exhausted memory.
DerivedSupported1 source— sign in to open

Sources

Publishers with included, body-backed reporting in this cluster.

devops.com
1 article · September 16, 2026
The Most Dangerous Reliability Failures Aren’t Component Failures

Ask Clarity

Have a question this piece left open? The answer comes from its reporting and claim ledger — nothing else.

Try

Clarity analysis

How the coverage leans and what the record actually supports. Read the methodology

Perspective Coverage

How much of the coverage speaks to builders, operators and investors. It measures emphasis, not whether the story is right.

For anyone who owns a shared library, this story is really about an unwritten assumption. The policy code loaded one storage instance's state into memory, computed the diff and applied it, which was safe because the only caller was an engineer at a terminal doing one instance at a time. The provisioning service then called the same function hundreds of times at once. Per-call cost never changed. The author also notes the library had been optimized shortly before the incident and tested against the serial pattern, so the benchmark covered the path that was already working, not the one that broke.

Reality

Evidence, adoption, hype gap, incentives and confidence, each read independently from the current record on the scale shown beneath it. How these are measured.

Evidence38

0100

One practitioner account, no figures

The failure is described consistently and in enough detail to follow: shared library, one bounded caller, one concurrent caller, memory exhausted at high fan-out. But every fact rests on the recollection of the engineer who wrote it up for devops.com, and the piece does not give the memory limit, the concurrency level or the size of the request that broke it. That leaves a causal story that sounds plausible but that we cannot verify.

AdoptionInsufficient

No deployment record

The account does not say whether the admission-control rule it proposes was implemented, and it names no system, team or release. There is nothing to measure uptake against.

Hype gap+22

Topics and entities

Follow any of these and your For You feed starts watching them — no settings page required.

Topics

Entities

product
Tiered autonomy lets AI handle proven incidents while humans approve or command the rest
1 publisher
product
Half the incident clock goes to search, and telemetry tools cannot read the answer
1 publisher
product
A senior stakeholder found the new tab on their own phone an hour after the flag flipped
1 publisher
product
OpenTelemetry is free; the collector fleet, the retention policy and the on-call rota are not
1 publisher

What happened

Why it matters

What to watch

Claim ledger

Sources

Let Clarity write the post for you.

Topics and entities

Topics

Entities

Related stories

Claim ledger

Sources

Let Clarity write the post for you.

Topics and entities

Topics

Entities

Related stories

Perspective Coverage

Reality