Three schemas, one shock absorber: put upstream drift in the adapter

A post on dev.to makes a narrow structural claim about document and retrieval pipelines: they contain three schemas that evolve independently, and conflating them is the root of most of the pain ^[1]. The reason to care is cost asymmetry. According to the post, the source schema is whatever upstream emits, you do not control it, and you are frequently not told when it changes ^[2]; the internal normalised document and chunk representation is fully yours and should change rarely and deliberately ^[3]; the index schema is the set of fields the search store knows about and can filter on, and changing it often means a rebuild ^[4]. One of the three is out of your hands and one of the three bills you in rebuild time ^[5]. That asymmetry is the whole argument for where tolerance lives. The post puts the shock absorber in the mapping from source to internal: one adapter per source, each responsible for producing a valid internal document or failing loudly, so the rest of the pipeline never sees an upstream change at all ^[6]. Upstream will add a field, rename a field, change a type from string to object, and start sending null where it never did; none of that is avoidable, and what is avoidable is finding out from a crash in a consumer at three in the morning ^[7]. The unpopular half of this is the constraint on the internal schema: it should contain nothing specific to one source, because the moment a field exists only because a particular vendor sends it, every other adapter has to decide what to put there and the internal schema has become a union of external ones ^[8]. The post's alternative is a namespaced extras map, available to anyone who wants it and mandatory for nobody ^[9]. The vocabulary for deciding whether a change is safe already exists. The post points at Avro's schema resolution rules and the compatibility modes used by schema registries, which give four named answers to whether a change can be deployed ^[10], and argues that the transitive distinction is the one that catches AI pipelines specifically, because reprocessing the archive is a normal operation rather than an exotic one; if you will ever re-embed three-year-old documents from stored artefacts, you need FULL_TRANSITIVE behaviour rather than FULL ^[11]. Underneath all four modes is one rule: a field added without a default is breaking, because a reader encountering old data has nothing to put there, and a default converts an incompatible change into a compatible one ^[12]. At the boundary the rules are mechanical. Unknown fields are captured rather than dropped and are not fatal, missing optional fields take defaults, and a missing required field fails that document and only that document ^[13]. The worked example uses a Pydantic model configured with extra="allow" so unknown keys survive ^[14], and an adapt function that catches ValidationError, quarantines the raw record with the error as the reason, and returns None for that record alone ^[15]. Unknown keys outside a known set increment a source.unknown_field metric tagged with the field names ^[16], which the post calls the cheapest early warning available: upstream adds a field weeks before anyone tells you, the counter moves, and you learn from a metric instead of from a consumer that broke when the field became mandatory ^[17]. The failure granularity is the part most teams get backwards. Failing the batch on any invalid record sounds rigorous and behaves badly, because one malformed document out of two hundred thousand stops the pipeline, and the pressure to restore flow produces a bypass flag that then stays on forever ^[18]. Per-document quarantine with a visible count carries the same information without the outage ^[19].