The two identifiers do the work. Nylas delivers webhooks at least once, so a redelivery must not spawn a second shadow run [5]. The top-level notification id is the delivery key and the inner `data.object.id` is the message key, and the post keeps both because they answer different questions [6]. `shadow_events` makes the delivery key its primary key [7]. `shadow_runs` keys on `(notification_id, candidate_version)` [8]. The dedupe unit is therefore delivery times version: a redelivered notification for the same candidate collides and does nothing, while running prompt version A and model version B against the same notification inserts two rows instead of pretending they are one experiment [20][10].
Order matters more than the DDL. Insert the event and its pending run inside the webhook transaction, before enqueueing, with `ON CONFLICT DO NOTHING`, and enqueue only when that statement returns a row [9]. Put the enqueue first and you have moved the idempotency requirement into the worker, which is the half that pays for the model call [17].
Count the columns in `shadow_runs`. Eleven, of which three hold what the candidate produced (`proposed_action`, `input_hash`, `input_snapshot`) and three hold what it gets measured against (`outcome`, `reviewer_verdict`, `actual_message_id`) [18]. That ratio is the honest cost of the stage. Proposals are cheap; the author is clear that the record is only useful beside an outcome you already trust, meaning the production agent's action, a human-approved reply, a support label, or an explicit reviewer verdict [13]. That is also the condition for this design transferring to your shop. If support is humans today and nobody writes the action taken into a structured field, there is nothing to join against, and shadow runs pile up as ungraded prose.
The failure mode named in the post is precisely the one a prose review misses: a candidate that writes well and would have answered a billing question instead of escalating it [14]. A tool change that quietly makes the agent write twice is the same shape [21]. Comparison catches both. Reading the draft catches neither. And the `confidence: 0.91` in the sample proposal is a field the model emitted, not evidence about the model [12].
The draft rule is the sharpest line in the piece: do not create a Nylas draft for the candidate reply, because drafts are visible mailbox state and a human can mistake an experiment for a proposed response and send it [11]. A draft is an experiment with a Send button next to it.
Retention bounds what you can learn later. The example keeps an encrypted, access-controlled input snapshot for 30 days [16] while `input_hash` stays in the row permanently [8], so once the snapshot expires you can still tell whether two runs saw identical input, but you cannot replay either one [19]. The author also declines the easy claim that no-send equals safe: the candidate still reads live customer content, so run it only where you already have authority to process that content, and leave attachments and tools out unless the experiment needs them [15].
