Build1 publisher3 min readPublished
Anthropic's Batch API returned nothing usable for 112 of one developer's 1,842 requests
One operator pulled 30 days of logs and found 71 errored, 24 expired and 17 truncated results sitting behind a batch status that read "ended" every time. Matching by position had already mis-filed reports.
The Engineer · Build desk
What happened
- A developer running a mock-interview platform pulled 30 days of logs and counted 1,842 scoring requests sent through the Anthropic Batch API in 96 batches, of which 1,730 came back usable.
- The other 112 requests split into 71 that errored, 24 that expired at the 24-hour processing wall, and 17 that the API marked succeeded while returning a truncated JSON body.
- Because results arrive in arbitrary order and the code matched them by position, a single errored request shifted every later entry up one slot and attached one candidate's report to another candidate.
- The batch discount halved the 30-day bill from $209 to $104, with median latency of 9 minutes and a p99 running past 6 hours.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A result the API labels succeeded can still be half a JSON object, so "the call returned" cannot be the success condition for any step that parses the body.
- exposure Position-matched results put one user's content into another user's report without raising anything, which leaves the customer as the detection mechanism.
- decision With a p99 past 6 hours, the choice of what goes asynchronous has to be made per endpoint: anything a person waits on stays synchronous even when the prompt looks batchable.
- cost At this volume the discount is worth about $105 a month, so the reconciliation and null-row alerting work has to cost less than that before halving the inference bill is a win.
A 6.1% silent failure rate is one account's number, over 30 days, on one model [2]. The composition is more useful than the percentage. Of the 71 errored requests, 52 were `overloaded_error` [5]. That is capacity on Anthropic's side, and it moves with their load. Fourteen were `invalid_request_error`, from interview transcripts that blew past the developer's own token budget [5]. The 17 truncations were responses the API called `succeeded`, carrying `stop_reason: "max_tokens"` against a `max_tokens` of 4096 [6][7]. The four failure classes transfer to anyone. The rate transfers only to a pipeline with similar prompt lengths and a similarly tight output ceiling, hitting the same capacity at the same hours.
Results come back in arbitrary order [8], and the original loop matched them to the input list with `zip(pending_sessions, results)` [9]. That was the expensive bug, and it produced no missing row at all. When every request succeeds, the order is often close enough to submission order that nothing looks wrong [10]. One error shortens the results list by one, and every entry after it shifts up a slot [10]. Candidate A gets Candidate B's report, rendered in the template, with nothing raised [10][8].
The developer wrote that a batch ending and the requests succeeding are two completely different facts, and that the code had been written to assume they were the same one [21].
`ended` means Anthropic has finished working on the batch, including the parts it gave up on [11]. That is all the flag says. Per-request outcome lives in `batch.request_counts` and in each line of the JSONL results stream, where `result.type` is one of `succeeded`, `errored`, `expired` or `canceled` [12]. On an errored or expired entry, `entry.result.message` does not exist, so those lines raised an `AttributeError` that was being caught too broadly [13]. On a truncated body, `json.loads` threw, the `except` block logged at DEBUG, and the row stayed null [14]. The first one surfaced when a user emailed a screenshot of an empty report [22].
The discount is the reason to tolerate any of it. The 30-day bill went from $209 to $104 [15], a saving of $105 [2], or about 5.7 cents per request across 1,842 requests [3]. Those requests went out in 96 batches [1], an average of about 19 per batch [1], against a documented ceiling of 100,000 requests per batch [16].
Median latency was 9 minutes and p99 ran past 6 hours [17], at least 40 times the median [6]. Placement is the other decision. The nightly portfolio scoring pass fits that profile: nobody is watching and the prompt is fat [19]. The interview report does not, and the developer batched it anyway, calling it the first mistake [18].
Two changes cover most of the gap: index every result by its `custom_id` and write only what you can match, gating the write on `result.type` [12][8]. Then treat `stop_reason: "max_tokens"` as its own failure class, because the API reports it as success [6]. One more thing from day one: `custom_id` comes back in the results file, so keep it opaque, and the developer's warning is not to put an email address there because it was convenient [20].
What to watch
- Whether Anthropic publishes overloaded_error rates for batch traffic, since 52 of the 71 errors came from its capacity rather than the caller's requests.
- A follow-up count from the same pipeline after custom_id reconciliation, which would separate real API failures from the matching bug.
- Any SDK change that exposes per-request outcomes without the caller reading and joining the JSONL stream by hand.