Build1 publisher3 min readPublished
Four of the eight published compatibility checks can fail while the endpoint returns HTTP 200
The eleven-check suite comes from a gateway maintainer who wrote it to be pointed at their own endpoint as well as everyone else's, and the checks that matter most are the ones where a divergence still answers with success.
The Engineer · Build desk
What happened
- A gateway maintainer writing on dev.to argues that "OpenAI-compatible" was never a boolean but a surface area, with every implementation covering a different subset of it.
- The tool-calling check notes that function.arguments is a JSON-encoded string, so an endpoint that helpfully returns a parsed object breaks every client calling json.loads on it.
- The same check asks whether finish_reason comes back as tool_calls rather than stop, because agent loops branch on that value.
- logprobs and top_logprobs are named as the parameter a proxy layer usually drops first, on the grounds that almost nobody notices it is gone.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost When a shim implements stop by post-truncating the completion, the tokens you never see still land on the invoice, and the usage field is the only place the overspend shows up.
- constraint Confidence gating and probability-comparison classifiers narrow the set of endpoints you can route to at all, because a proxy that quietly drops logprobs removes the signal they run on.
- decision An endpoint that accepts json_schema and ignores it fails roughly one request in fifty and presents as a model quality problem, which sends the fix toward a model swap instead of a schema validator.
That first failure in the post is a system, not an anecdote. Arguments for a streamed tool call arrive split across chunks the accumulator did not expect, the JSON parser throws inside a retry loop, and the retry loop then hammers the endpoint because the error body does not carry the field the backoff code reads [2]. Two separate conformance gaps compose, and neither is an outage on its own; a retry loop that cannot read an error body is a load generator.
The checks to run first are the ones where a divergence still answers 200. Four of the eight described do that: `response_format` accepted and ignored, which returns prose with a code fence around it [8]; `temperature` accepted and ignored [10]; `stop` implemented by post-truncating the full completion [11]; and an error object served under a 200 status [13].
Three of the eight only misbehave when `stream=true`. The delta sequence itself is one: the reference has a first chunk carrying `delta.role = "assistant"`, content fragments in the middle, `finish_reason` on the last content-bearing chunk, and a literal `data: [DONE]` line to terminate, and implementations diverge on all four points [6]. Tool-call fragment reassembly is the second [3]. Usage accounting is the third, since it appears only if you pass `stream_options: {"include_usage": true}` and then lands in a final chunk with an empty `choices` array [12]. A green run collected without streaming says nothing about any of the three.
The reassembly check needs care in how you probe it. An endpoint that re-sends `id` on every fragment, or omits `index` when there is only one call, works with a naive accumulator and fails the moment a model emits two parallel calls [3]. A probe that asks for one tool call cannot see that. Ask for two.
Two limits in the material. It is billed as an eleven-check suite, the available text enumerates eight and breaks off inside the eighth [15], so the remaining three checks and the script cannot be read. And the eight that are enumerated all sit in chat completions parameters and response shapes [3], which is the surface most clients hit and not the whole protocol.
The disclosure is doing real work here. The author maintains daoxe, a multi-model gateway that speaks the OpenAI protocol among others, states the checklist is written so you can run it against them as well as anyone else, and says a check that makes them look bad is the correct output [14]. The only suite worth recording results from is one blind to the vendor name.
If you never write the harness, record the cheap version: per endpoint and per mode, which of the three `temperature` behaviours you are on. Reject, ignore, and honour are all defensible. Not knowing which one you have is the failure [10].
What to watch
- Whether the three checks missing from the available text leave the chat completions surface, which decides how much of the protocol the suite actually covers.
- Whether the endpoint-blind script lands as a runnable artifact with published per-endpoint results rather than a prose checklist.
- Whether any gateway operator publishes its own failing checks, which the daoxe author explicitly invites.