Skip to content

Build1 publisher2 min readPublished

DumboDB turns MongoDB oplog events into per-field commits from a hidden replica member

DoltHub's DumboDB now takes its version history from writes an application is already sending to MongoDB. DumboDB joins as a hidden member the cluster cannot elect, and the first release runs only with the security features switched off.

The Engineer · Build desk

Illustration accompanying DumboDB turns MongoDB oplog events into per-field commits from a hidden replica member

What happened

  • DoltHub said on September 22nd that its DumboDB document database can join an existing MongoDB replica set, consume the changes it sees and turn them into a commit history.
  • DumboDB has to join as a hidden, non-voting member with priority zero, and it refuses to replicate unless all three of those settings are in place.
  • DoltHub's repository labels DumboDB pre-1.0 and not ready for production use.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint V1 can only attach to a cluster running without authentication or TLS, so the commit history an operator can get today comes from test writes.
  • decision Trying it means reconfiguring the source replica set to add a member, so the team that wants the change history needs sign-off from whoever owns the cluster's failover configuration.
  • capability Per-field change inspection becomes available without porting an application to a different database, because the history is built from the oplog the cluster already produces.
  • exposure A hidden member consuming the oplog copies every write in the replica set into a second store, and in this release that link runs unauthenticated and without transport encryption.

Hidden, non-voting and priority zero each do a different job in a MongoDB replica set, and MongoDB's own documentation treats them as distinct roles [6]. Stacked on one member, they keep DumboDB out of the failover path. It reads the oplog, cannot become primary, does not participate in elections and does not count toward a majority write acknowledgment [4]. A member that is not counted in the majority cannot delay a `w:majority` write when it falls behind [21]. Refusing to start without all three settings is the right default [5].

DumboDB stores the change itself, with the field path and both values. In DoltHub's demonstration two documents are inserted into MongoDB and one document's `qty` field moves from 10 to 99, after which a query against DumboDB returns the updated document [7]. Its `dumboLog` command shows a separate commit for that update, including the field-level path `$.qty` and the old and new values [8]. A conventional replica keeps another copy of the database's state, and DoltHub is trying to make the path to that state inspectable [11]. The company built the same thing for MySQL, where its binlog replication guide records source transactions as commits [12].

Neil Macneale, the DoltHub engineer who wrote the replication announcement, says each replication event becomes a commit [9][23]. For that model to hold on a production cluster, the commit rate has to keep pace with oplog throughput, and the published example is two inserts and one update on MongoDB 8.0.28 and DumboDB 0.6.4, running on local addresses with authentication off [13]. The runtimewire account says the example does not establish performance under sustained write traffic [10].

Macneale writes that replication V1 does not work when MongoDB's `--auth` setting is enabled or over TLS, and he describes authentication and authorization support as a blocker for a realistic setup [14]. DoltHub's repository does document authentication for DumboDB's ordinary single-node mode, and that documentation stops there; the new replication path is not covered [15]. DoltHub has not presented the release as full MongoDB replica-set support [17].

DumboDB 0.1 shipped in May, and in September Macneale described the first phase of its clone, push and pull workflow [18]; the replication demo runs 0.6.4, five minor versions in roughly four months [22]. TerminusDB offers branching and merging for documents and knowledge graphs, and XTDB emphasizes immutable records and time-travel queries, according to the same account [20]. The lineage goes back to 2018, when co-founder Brian Hendriks put a Git-style command line on an existing version-controlled data project and the founders recognized the basis for Dolt, their MySQL-compatible database [19].

What to watch

  • A replication release that works with --auth enabled and over TLS, which is the limit DoltHub itself flags.
  • Any published commit-rate figure for DumboDB against sustained oplog traffic, instead of a three-write example.
  • The repository dropping its pre-1.0 and not-for-production labels.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories