A 200 Is Not A Row Count: How Agents Inherit Open Data's Silent Caps

The silence is structural. An HTTP status describes the transport, not the scope of the answer, so a truncated result set and a complete one arrive as the same 200 with the same content type, and nothing in the payload is obliged to say which one you got ^[1]. ArcGIS feature services do expose `resultOffset` and `resultRecordCount`, and may signal that a transfer limit was exceeded; Socrata supports paging and query controls ^[2][3]. Both are opt-in for the client. An agent that never asked for page two has no way to learn that page two exists.

That maps badly onto how agent plumbing handles errors. Retry and backoff exist for the loud failures. A cap fails at no layer the agent inspects: the JSON parses, the fields are the fields, the row count is plausible, and an empty page reads exactly like an answer ^[1]. The CivicDataForge team's version of the consequence is blunter than mine. Ignore the paging contract, they write, and "the source returned 1,000 rows" quietly becomes "the jurisdiction has 1,000 records" ^[4].

Their response is a five-state decision vocabulary in place of a Boolean: EVIDENCE_FOUND, NO_PUBLISHED_MATCH, REVIEW_REQUIRED, SOURCE_UNAVAILABLE, SCOPE_INCOMPLETE ^[8]. Count what that is admitting. Four of the five states report the condition of the query rather than the content of the world ^[1]. On a wired-up public data source, the honest output most of the time is not a fact about a permit or an address, it is a description of how much you were allowed to see. Their example is an address lookup that returns no permit row, where NOT_PERMITTED is usually stronger than the evidence: the publisher may not cover the whole legal regime, the address may be formatted differently, the permit may live in another system, the source may be stale, or the query may simply have been incomplete ^[7].

The same logic runs through their health checks, which is where I would put the money. Availability is one of seven monitored dimensions, alongside schema compatibility, record-count behavior, source date integrity, freshness, content fingerprint and contract compatibility ^[12][2]. A source can be reachable and stale, can hold a stable schema while returning incomplete data, can change content without changing its URL and change its schema without failing a request ^[11]. When freshness cannot be measured they emit `freshness_not_measured` rather than let a reachability check imply it ^[13].

The cost of this is real and it lands on the collector, not the consumer. You have to discover the publisher's page limit, request every page in a stable order, track page count against observed rows and any source-reported total, and refuse a monitoring baseline outright when the selected query came back capped or partial ^[5]. That refusal is the load-bearing part: a partial page is fine for exploration, but it cannot prove a record disappeared between two runs ^[6]. In their evidence envelope, `complete` is a boolean on the scope object rather than on the record ^[10], which is the correct place for it, because the thing you are uncertain about is the query, not the row.

One caveat on provenance: this is a single self-published account from a team selling the architecture it describes, offered as the failure modes they hit rather than proof that government data can be made clean ^[14]. The pagination mechanics are checkable against the platform docs. The rest is a design argument, and it is a good one.