Build1 publisher2 min readPublished
Three refresh paths turned one shared reference table into three different answers
A stale classifier changes what a balance means. Denis Toropov's account of a three-database balance system shows three copies of the same account types and statuses drifting out of agreement.
The Engineer · Build desk
What happened
- Toropov describes balance data split across three databases, one behind the account display service, one behind the service showing customer-level balances, and one holding current account balances.
- All three services depended on the same reference data, including account types, statuses, product attributes and classifiers.
- Users saw one account placed in one category on one screen, aggregated by a different attribute on another, and interpreted a third way somewhere else.
- For the team the divergence turned into incidents, manual reconciliation, long investigations and eventually a loss of trust in the data.
- According to the post's headline, the shared reference data was moved into a dedicated service.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Reconciliation work grows faster than the service count: three local copies give three pairs to compare, and a fourth copy gives six, because pairs are n(n-1)/2.
- decision No consistency guarantee can be written until the team decides which event counts as the moment a reference change is applied, and that decision has to precede any deletion of copies.
- exposure A single owning service becomes a dependency of every balance read path, so one reference lookup failure now reaches the account view, customer balances and current balances together.
Each of the three services refreshed the shared reference data its own way: one on a schedule, one reacting to an event, one through a separate integration flow [6]. The drift that followed was asynchronous, with one service having applied a change, another not yet, and a third updating through its own process [5]. Had all three been fed by a single publication path, they would lag together, and a consumer could at least ask how far behind it is.
The sharpest artifact in the account is the local mapping. One service introduced one as a temporary workaround, and it survived multiple releases [13]. That mapping makes the copy a fork, and nobody outside that service's code can see it.
Toropov wrote that "in practice, reference data defines how the system interprets the primary data" [14]. A balance is a number until product type, account status, segment membership and aggregation attributes give it meaning [16]. He calls the opposite habit a common design trap, where reference data is treated as secondary, so a local copy sits beside every service and gets synced somehow later [20]. The post lists the questions this leaves open, among them who is the source of truth for a reference entity and who owns the investigation when the same field is interpreted differently across systems [11]. Toropov wrote that "at that stage we did not have one good answer to those questions" [12].
Incidents of the form "why is it one thing here and another there?" required investigating three databases, three services, three update histories, and often several teams [7].
The supplied text of the post breaks off before the dedicated service is described [18]. In my view the work ownership has to do is narrow. The owning service is the only writer, and it publishes a version or effective date a consumer can quote in a ticket. The consumers keep their caches, because the three databases exist in the first place for different read patterns, different performance requirements and different representation models [3], and a network call per row does not change any of that.
The cost of the move is a new dependency. One service sits in the read path of the account view, the customer balance view and the current account balance, and its downtime becomes theirs. I would still take that trade here: Toropov's account reports a growing gray zone that lived as chronic operational pain, not a single major outage [8].
What to watch
- Whether the full post publishes the owning service's contract: versioning, effective dates, and what consumers may cache locally.
- Whether the three services keep local copies after the move, and which single refresh path replaces the schedule, event and integration-flow split.
- Whether the account display, customer balance and current balance split survives once every reference read crosses a service boundary.