Skip to content

Build1 publisher3 min readPublished

Pharos Production grades indexer reorg recovery against an independent replay of the chain

Pharos Production's SQLite rehearsal passes an indexer only if its post-reorg state matches a pure replay of the chosen branch, a total of 24 at block B4. It also reruns and crashes recovery, since a caught-up checkpoint can still sit on an abandoned branch.

The Engineer · Build desk

Photograph accompanying Pharos Production grades indexer reorg recovery against an independent replay of the chain
Photo: dev.to

What happened

  • An indexer can advance its checkpoint and keep a green health endpoint while its token balance still counts a transfer that is no longer in canonical history.
  • Pharos Production published a Python and SQLite model that indexes signed amounts, swaps out an orphaned suffix and checks the result against a separate pure replay.
  • In its fixture the abandoned branch totals 15, rolling back its suffix returns the aggregate to 10, and the replacement branch ends at 24.
  • The model uses synthetic block IDs and pre-selected branches, and it does not implement Ethereum consensus, validate block hashes or connect to a node.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • exposure A wallet can show users a wrong balance while every smart contract executed correctly, so reorg recovery has to be tested in the application's own acceptance criteria.
  • decision Teams have to settle which outputs users rely on, wallet balances or analytics history, before choosing a recovery design, because those outputs are what the test grades.
  • constraint A pass on one chain and one aggregate leaves production indexers with several projections and concurrent readers untested until the assertions are adapted to them.

The failure starts in the checkpoint. According to the tutorial, a checkpoint that stores only a height cannot identify a chain, because two competing blocks can occupy the same position [8]. The fix it proposes is to record the block number together with its hash, and to keep enough parent relationships to establish which stored blocks belong to the replacement branch [8].

Event rows need the same care. Pharos proposes an occurrence key of chain identity, block hash, transaction hash and log index [9]. A transaction hash alone cannot tell apart the same transaction's appearances across competing histories [9]. The node will not deduplicate for you. The Geth documentation states that "a subscription can emit logs for the same transaction multiple times." [10] The tutorial notes this describes notification behavior and does not promise that a disconnected consumer receives every correction [11]. When a transaction lands in a different block, its output has to be rebuilt from the new receipt [11].

The fixture is small enough to audit by hand. The shared prefix is G to A1, the abandoned suffix is A2 to A3, and the replacement is B2 to B3 to B4 [12]. An indexer that applied the replacement but kept the orphaned -2 event would report 22 against a correct 24 at B4 [14] [4]. The replacement suffix happens to be one block longer than the one it displaces [19]. The model still never lets a branch win for having more blocks; the caller selects it [12]. In a model with no node attached, I think the caller is the right owner of that choice [6].

The grader is a separate pure replay function [3]. It has no tables of its own to corrupt. The two extra runs catch different bugs. Feeding the same input again must change nothing [5]. That run catches an indexer that double-applies a reorg it already handled. Throwing an exception mid-recovery must leave the previously committed state intact [5]. That run catches a swap that removes the old suffix and commits before the new one is written.

Pharos writes the recovery contract in terms of visible state. Canonical block membership, active event rows, materialized aggregates and the checkpoint must agree at a committed boundary [15]. Orphan records kept for audit have to be marked and excluded from current product queries, and deleting them is only one storage policy [15]. A single database transaction lets a reader see the old committed state and then the new one, subject to the database's isolation behavior [16]. An asynchronous projection pipeline needs a visible generation or watermark instead [16]. Without one, the tutorial warns, an API can combine the new event table with yesterday's aggregate and return a result that belongs to neither branch [16].

The passing run covers this fixture only. Its blocks carry synthetic identifiers, its branches arrive complete and already selected, and nothing validates a hash or connects to a node [6]. In my view, for the pass to transfer, a production indexer has to take its branch choice from a real node, check hashes before trusting parent links, and still match an independent rebuild of the selected canonical history [2]. Pharos Production says the tutorial does not claim a measured customer outcome [20].

What to watch

  • A version of the rehearsal with multiple projections and concurrent readers, the production case the tutorial leaves for adaptation.
  • A run against a live node, where branch selection and hash validation come from the client and not from the caller.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories