Skip to content

Build1 publisher2 min readPublished

A dependency walk that stopped at the first level of nesting filled a foreign-key column with NULLs

In the alpha migrator mongopg-migrate, dependency ordering and structure validation both stopped at the top level of explode, so a lookup nested one level deeper loaded backwards and a new on_missing policy wrote NULL over every row.

The Engineer · Build desk

Illustration accompanying A dependency walk that stopped at the first level of nesting filled a foreign-key column with NULLs

What happened

  • In mongopg-migrate, an alpha migration tool at v0.2.0, a field at any nesting level can be declared a lookup, resolved against another entity's already-migrated rows and rewritten as a foreign key.
  • The function that orders entity loads and the function that validates config structure both walked only the first level of explode, so a lookup nested one level deeper was invisible to both.
  • With the lookup at the top level the load order came out ['zcategories', 'hospitals']; moving the same lookup one level deeper flipped it to ['hospitals', 'zcategories'].
  • A new on_missing policy had just replaced the unconditional LoadError that an empty id_map used to raise, with on_missing: null writing NULL and letting the migration finish.
  • The run ended with a whole foreign-key column silently NULL while the count diff, the dangling-reference check and the structural validation all reported success.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint A structure validator built on the same walk as the resolver it guards cannot see what the resolver misses, so a second check written against the same config model buys agreement.
  • decision Anyone shipping a lenient on_missing default has to decide what it means when the target entity has zero rows, because treating that as ordinary dangling data converts a load-order bug into clean output.
  • exposure Three checks agreed on a column that was NULL for every row, so the first person to notice the damage is whoever queries that relationship weeks later.
  • cost The guard costs one indexed query per lookup entity, charged the first time a lookup to it misses, so the bill scales with the number of entities in the config.

The referencing entity gets scheduled first, `zcategories` has not loaded, and its id_map is empty, so every lookup against it misses [10]. `on_missing: null` was written for the ordinary case where a source document was genuinely deleted [9]. It cannot tell that case from a target entity that has never loaded a single row, and the author is explicit that those are different failure modes [14].

Nothing downstream can see the difference. NULLs do not change row counts [11]. The dangling-reference check runs after the full migration, and by then `zcategories` has loaded, so it re-checks the references and finds nothing wrong [12].

It took both changes for the run to finish without an error. An empty id_map used to raise an unconditional `LoadError` and kill the run [8], so the same nesting-blind ordering fault would have been loud one version earlier [1].

The root-cause fix is one line of recursion, with `_explode_lookup_targets` unioning in its own result for each nested `explode` [15]. When `on_missing` is anything other than `ERROR`, the loader asks whether the target entity has any id_map rows at all, and raises `LoadError` when the answer is none [16]. Its message tells the operator to go looking for a load-order bug or a forgotten prerequisite run instead of a genuinely dangling reference [17]. Caching makes it cheap: one extra indexed query the first time a lookup to that entity misses, not one per row [18]. It also catches a prerequisite migration run that nobody ever executed, and correct ordering does nothing for that one [19].

The author first wrote this up as a one-off, and says two bugs since have changed that view [24]. "Two checks aimed at one mistake are one check," the author wrote [20]. The `--pg-schema` flag is the second case. It exists so you can migrate into a schema other than `public`, and it reached `introspect_postgres()` and nothing else [21]. Every write named its table unqualified, so rows landed wherever the connecting role's `search_path` pointed, normally `public`, and nothing errored [22]. Then `validate` counted those same wrong tables and printed `[OK] hospitals (hospitals): mongo=4002 postgres=4002` [23].

This is one developer's report on an alpha tool at v0.2.0 [1], and the two orderings are a repro, not a measured population [6]. The shape reaches your pipeline if the code that orders work and the code that validates configuration derive their view of the schema from the same traversal. A level that traversal never walks is then invisible to both, which is why a typo'd entity name in a nested lookup came back from validation with zero issues [7].

What to watch

  • Whether the recursive _explode_lookup_targets and the has_any guard land in a tagged release after v0.2.0.
  • Whether --pg-schema is carried into the write path and the validator instead of only introspect_postgres().
  • Whether the has_any guard needs an exemption for entities that legitimately migrate zero rows.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories