Skip to content

Build1 publisher3 min readPublished

ChaosCypher's graceful shutdown wrote the one task status its own requeue script refuses to revive

Version 0.4.3 of ChaosCypher ships 54 commits and no features. Four of the fixes are queue recovery paths that held in the ordinary case and lost or duplicated work once a worker stalled or was told to stop.

The Engineer · Build desk

Illustration accompanying ChaosCypher's graceful shutdown wrote the one task status its own requeue script refuses to revive

What happened

  • ChaosCypher v0.4.3 ships no new features, no breaking API changes and no schema migrations: 54 commits since v0.4.2, 23 of them changing shipped code, most in the queue and the extraction pipeline.
  • On shutdown, the drain's CancelledError arm wrote terminal status="cancelled" to the tasks it interrupted, which is exactly the status the requeue script and retry_task refuse to resurrect.
  • A max_tries=5 task with retry_on_crash=True got three dispatches instead of five, because requeue_atomic.lua bumped attempts on top of the worker's own claim-time increment.
  • MCP add_document read file_path from disk even when content was supplied, and a caller passing both got an uncontained read that reached /data/credentials.json.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • decision Operators still on earlier releases have to pick a shutdown signal on the strength of this defect, and the hard kill is the one that leaves work recoverable.
  • cost Every duplicate delivery before this release paid for the same chunk's LLM call twice, and the operator absorbed that spend with nothing raising an error.
  • constraint The four crash-retryable operations with no owning resource had no second layer to make up the missing dispatches, so the accounting bug cut retries for exactly the calls that depend on the queue.
  • exposure A read-only configuration bounded an external assistant only when the flag was typed, so the exposed installs were the ones launched by a client's own command line.

Redis EXPIRE is a silent no-op on a key that has already expired, and refresh_heartbeat treated the call as a successful refresh [8]. A worker stalled for 31 seconds could never get its lease back, so the reconciler condemned a multi-hour task as abandoned and requeued it for duplicate execution [9]. The replacement is a SET with EX, which writes the key back into existence instead of extending one that is gone [8].

The extraction short-circuit failed in the same place. Re-running a chunk was safe because the handler looked for a completed row first, and a row still in running went straight through [10]. Two deliveries produced two terminal writes and two job-counter bumps, and the job could finish one chunk early [10]. Both transitions are now guarded updates that return a rowcount, and the handler bails out before the billable LLM call when it loses the claim [11].

Two units of budget per crash cycle against a five-unit limit leaves three dispatches, so 40 percent of the promised retries never issue, while the terminal message still reports "crashed after 5 attempts" [21][6]. The operations affected are the four crash-retryable ones with no owning resource: execute_workflow, execute_step, export_graph and export_by_sources [7].

The extraction fixes touch the part of the product that invites inspection. Every relationship ChaosCypher extracts carries a justification, the evidence the model gave for linking two entities [12]. One parent_of edge at confidence 0.9 explained itself with "I will link 9 to 3 via interacts_with" [13]. That string is the model planning its next step, stored in the field a reader opens to check the edge. Justifications carrying reasoning markers are now blanked, and the rest are trimmed to two sentences on a sentence or word boundary [14].

Three more defects came from reading the wrong variable. The chaoscypher mcp guard read the --mode flag instead of the effective setting, so the documented Claude Desktop invocation against a read-configured install still handed the external client the destructive apply_upgrade tool [16]. CHAOSCYPHER_ALLOW_USER_PLUGINS=0 is documented as the Docker kill switch for user plugins, and no service declared it, so the loader always saw its default [17]. list_triggers deferred the filters and actions columns the dispatcher reads, and every configured trigger filter matched everything [18].

Each of these paths behaved correctly in the state its author had in mind. The failures sit at transitions: a shutdown, an expired key, a redelivery, a launch without a flag. A test suite that never asks a worker to stop mid-task will not find the cancelled-status defect, and the release notes describe the whole class the same way, as work lost, counted twice or quietly downgraded with nothing reporting it [19]. The notes do not say when any of the defects were introduced [20].

What to watch

  • Whether v0.4.4 adds a shutdown test that interrupts a worker mid-task, the case that produced the cancelled-status defect.
  • Whether the requeue script learns to tell an operator cancellation apart from a drain-interrupted task, or the drain stops writing a terminal status at all.
  • Whether other MCP servers repeat add_document's pattern of trusting a schema statement that no code enforces.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories