A dev.to write-up on production experiment cost attribution argues that the emergency stop for an expensive AI feature belongs in a small control plane, that every worker should gate the costly path on a locally cached flag, and that cost should be attributed by tenant cohort, flag revision, and evaluation version [1]. The reason that third part matters is that a rollback which halts spend but leaves you unable to say which cohort spent what, under which revision, produces a quiet dashboard and no answer. The proposed shape has three pieces: an authenticated API that owns flag state and a monotonically increasing revision, application workers that poll or subscribe, cache the last valid state and gate work locally, and logs and cost events that carry the decision context [2]. The stated goal is to keep the emergency action separate from deployment while retaining enough context to ask later whether the experiment improved outcomes at an acceptable cost per cohort [3]. The post is precise about the word "new". A kill switch can stop new evaluations entering a costly path; it cannot pretend already accepted work never existed, so queue consumers, retries and streaming responses need an explicit drain or cancellation policy rather than one Boolean carrying every meaning [4]. That is the failure mode most teams ship: a single environment variable that means admission, cancellation and accounting all at once. Rollback is therefore modelled as a state transition with an audit record, not an environment-variable edit. The write carries the desired state, the revision the operator observed, an actor from the trusted identity layer, and a reason [5]. If another operator has already moved the flag, the API returns 409 Conflict instead of overwriting newer intent [6], and the read path returns the current revision so workers can reject stale updates and operators can confirm convergence [7]. The hot path reads from memory, because calling the control API on every model request adds a dependency exactly where predictability is needed; a background task refreshes the cache and the request handler reads it [8]. Startup default and stale-state policy follow the feature's risk: an optional experiment can default off, while a workflow whose abrupt removal would corrupt state may need a separately designed degraded path [9]. The honest caveat is stated plainly: a cached decision propagates only as fast as the refresh mechanism, so the runbook needs a measured propagation objective, not a promise of an instant switch [10]. Two clocks make that objective auditable. The control plane records when state changed; each worker records when it observed the new revision [13], which means a single "flag flipped at" timestamp cannot demonstrate propagation on its own [19]. During an incident you chart evaluations by revision and expect the old-revision line to stop accepting new work inside the declared objective [14]. In-flight work may finish under the old revision, and its event must keep that revision, because rewriting it damages the comparison [15]. Scope discipline holds the whole thing together. One key, such as cohort-cost-guard, with state that says whether the experimental path is on, and tenant assignment left in the experiment service where it can be deterministic and testable [11]. An ever-growing tenant list inside the emergency flag turns a rollback control into a second segmentation system [12]. The accompanying example is standard-library Python with an in-memory store, one read and one conditional update, and the author says it is unsuitable for multiple processes or hosts [16]; production needs atomic compare-and-set, durable audit history, authentication at the boundary and transport encryption [17]. The demo seeds revision 7 with actor release-automation [18], which is the right hint: automation writes to this thing too, and it should conflict like anyone else.
