Skip to content

Build1 publisher2 min readPublished

A timestamp in the system prompt turns prompt caching into a 25% surcharge

Prompt caching keys on the exact bytes of the rendered prompt up to each breakpoint, so a volatile string sitting before your last one makes every turn write a fresh entry that nothing will ever match.

The Engineer · Build desk

Illustration accompanying A timestamp in the system prompt turns prompt caching into a 25% surcharge

What happened

  • Turning on prompt caching for an agent loop that resends a 12K-token system prompt every turn raised the bill by roughly a quarter.
  • The usage object on each request reported 12,184 cache creation tokens, zero cache read tokens, and 291 uncached input tokens.
  • The volatile bytes came from a timestamp injected into the system string by a logging helper written months earlier, three call frames from the request.
  • The post names usage.cache_read_input_tokens as ground truth: zero across repeated requests with identical prompts means something upstream is rewriting the prefix.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • exposure Every request returned 200 and latency was unchanged, so an error-rate and p95 dashboard cannot see this failure at all; it appears only in the per-request usage object.
  • cost The premium is charged per turn to whoever owns the inference budget, and this loop paid it 40 times in a row with no failed request to appeal to.
  • constraint Byte-identical payloads can still miss, on the 20-position lookback in long tool turns and on parallel fan-out where an entry is readable only after the first response starts streaming, so a passing two-call check proves less than it appears to.
  • decision Anyone running a long-lived agent loop has to decide whether an assertion on cache_read_input_tokens belongs in CI, because the costly version of this bug arrives months later when someone appends a feature flag to the system prompt.

The API renders tools, then system, then messages, hashes the byte stream up to each `cache_control` breakpoint, and looks for an existing entry [5]. One changed byte at position N invalidates every breakpoint at or after N, and everything after the first divergence is cold [4]. The dev.to post files two more shapes of the same bug alongside the dynamic header [12]. `json.dumps` without `sort_keys=True` serializes the same dict with a different key order; the dict is unchanged and the bytes are not [12]. A tool list built per user renders at position 0, so nothing caches across users [12].

Writes bill at 1.25x base input price, 2x on the one-hour TTL, and reads run about 0.1x [6]. The post works the pair: two requests sharing a prefix cost 1.25 + 0.1 = 1.35 against 2.0 for two uncached sends, and it labels that break-even [7]. On those same figures the pair is 32.5% cheaper, since (2.0 - 1.35) / 2.0 = 0.325 [2]. A write that is never read is 1.25x for nothing [6].

The author wrote that a cache that only ever writes "is not a cache, it is a 25% surcharge" [16].

In a working multi-turn loop, `cache_read_input_tokens` covers the whole prior prefix and grows turn over turn [11]. `cache_creation_input_tokens` stays small, roughly last turn's output plus the newly appended input, because writes bill only the delta past the highest hit [11]. `input_tokens` is the tail after the last breakpoint [11]. Total prompt tokens are the sum of the three, so an agent that ran for an hour reporting `input_tokens: 4200` is neither cheap nor broken until you add the fields [10]. When creation sits near full conversation size every turn, something upstream is rewriting the prefix [11].

The increase is a measurement of one loop. For a quarter to be your number, uncached input has to be most of your bill, and the stable prefix has to be almost all of each prompt: here the write-rate tokens were 97.7% of the 12,475 prompt tokens per request [1]. Where the stable prefix is a smaller share of the request, the same broken prefix still bills 1.25x on everything it writes, and the surcharge lands proportionally smaller [6].

What to watch

  • Whether the provider documents the 20-position lookback window and the fan-out readability rule; the post asserts both without pointing at a spec.
  • Whether the one-hour TTL at 2x on writes ever pays back for a loop whose turns are minutes apart.
  • Per-request logging of all three usage fields, so a prefix break shows up as a trend before it shows up on an invoice.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories