Skip to content

Build1 publisher3 min readPublished

A generation check before the write drops the retry that re-granted access

A provisioning worker woke from a timeout and put a deactivated account back in a group, minutes after the audit trail logged the removal. The fix gates each job on a counter the identity record owns.

The Engineer · Build desk

Illustration accompanying A generation check before the write drops the retry that re-granted access

What happened

  • Access was removed from a deactivated account and the audit trail logged the removal as successful, and a few minutes later the group membership was back with no administrator involved.
  • A worker had sent the group addition, the destination accepted it but the response timed out, the user was deactivated in the authoritative identity system, and the woken worker retried the old addition.
  • The proposed worker loads the authoritative record for the job's subject and returns "superseded" when the job's generation is lower than the record's, reconciling against current desired state instead.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • cost Every access-changing job pays one read of the authoritative record before it writes, and the identity store has to own a monotonic counter that something increments on each decision.
  • decision Teams that delete the subject record on deactivation have to pick a retention window instead, because the tombstone is the component that rejects the late event.
  • exposure Dashboards built on connector success rates cannot see this class of failure, so departed accounts stay reachable while the provisioning graph reads green.

The gate is one comparison. The worker loads the authoritative record for the job's subject and returns "superseded" when the job's generation is below the record's [8]. In the worked example the job carries generation 41 for subject user-1842, and the deactivation has already moved the record to 42, so the grant never reaches the destination [6][7]. The job can still be acknowledged and recorded [9].

Timestamps were the obvious alternative and the post rejects them, because clocks and delayed producers make ordering messy [25]. "I prefer a generation assigned by the authoritative identity record," the author wrote [5].

What happens after the check passes is the more useful half. The worker does not apply the job's requested_change; it calls reconcile against the desired state it just read [8]. That desired state is a frozen record of subject_id, generation, an active flag and a group set [10]. The destination is read, compared, and moved by the smallest safe correction: no second create if the timed-out request actually landed, and a removal if the identity is now inactive [11].

This is why a dedupe key alone would not have held. An idempotency key answers whether the worker already did this. It says nothing about whether the instruction is still the right one, and the timeout told the worker nothing either, since a client can time out after the server has committed the write [12].

The cost is a monotonic counter on the authoritative identity record, one read of that record inside every access-changing job [8], and a retention rule. Deleting the subject at deactivation removes the thing that rejects the late event, so an inactive record or tombstone is kept long enough to reject old events [18]. Deactivation also gets reserved capacity, on the argument that creation is prioritized because delayed onboarding is visible and delayed removal is quieter [15][18]. Nobody files a ticket because their access failed to disappear.

The priority function makes the ordering explicit: 0 for an account still active in the destination but inactive in desired state, 10 for a missing account that should exist, 20 for everything else [16]. Removal work sorts 10 places ahead of onboarding and 20 ahead of ordinary drift [17].

For the mechanism to transfer, two things have to be true of your destination. It has to be searchable by an immutable external identifier, which is what the create-recovery path uses before it attempts a second create [13]; email does not qualify, because it can change and may be reassigned [14]. And the authoritative record has to be the only writer of consequence. A membership a person changes directly in the destination arrives with no generation to lose against, and the post lists that case, along with schema rejections and two identity sources disagreeing about ownership, among the failures that do not improve with time [19]. Those stop after a bounded number of attempts and land in a quarantine record carrying the desired state and the latest observation [20]. "An infinite retry loop is not resilience. It is a way to hide a decision the system cannot make," the author wrote [21].

The account is the second part of a dev.to series on security design meeting production systems [24], and it carries no measured numbers: no drift figures, no queue depths, no incident count. What it does carry is a claim about instrumentation. API success rate is a weak health metric here, because a connector can report successful calls while the wrong accounts remain active [22]. The post asks instead for the time destinations take to match desired state, deactivation drift kept separate from ordinary profile drift, and the age of the oldest unresolved high-risk identity rather than an average [23].

What to watch

  • Whether provisioning connectors start carrying a subject-scoped generation in the job payload by default, instead of each team bolting one on.
  • Later parts of the series putting numbers to quarantine volume or deactivation drift, since this account reports none.
  • Whether destinations expose lookup by immutable external identifier, since the create-recovery path depends on that search working.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories