Build1 publisher3 min readPublished
Fifteen deliberate primary kills put Redis Sentinel failover at down-after plus two seconds
A Docker lab that killed the primary fifteen times across Redis 8, Valkey 8 and Dragonfly found the outage tracking its down-after-milliseconds setting plus about two seconds of quorum vote and promotion.
The Engineer · Build desk
What happened
- A dev.to write-up killed an in-memory database primary fifteen times, across three down-after-milliseconds settings and three engines, Redis 8, Valkey 8 and Dragonfly, timing how long writes failed.
- The setup was one primary, two replicas and three sentinels under Docker, with the outage measured from the kill to the first successful write on the promoted replica.
- The author reports the outage tracking down-after plus roughly two seconds of quorum vote, leader election and promotion, and calls the relationship almost boringly linear.
- With down-after set to five seconds, Redis 8, Valkey 8 and Dragonfly all returned to accepting writes in 7.3 to 7.4 seconds under identical sentinels.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision A team weighing a Valkey or Dragonfly migration to shorten outages is tuning the term the measurement holds nearly constant; the argument belongs in the sentinel config file.
- constraint Sub-second recovery is not on offer from this design: even a one-second timer leaves about 3.3 seconds of failed writes, because the election and promotion cost does not shrink with it.
- exposure In the single-Redis Laravel shape, every app instance stalls in the same window: cache reads land on the database, sessions are gone, and Horizon cannot read or write jobs.
- cost Buying this failover behaviour costs three or five sentinel processes to run and monitor, plus a client that resolves the current master from a sentinel on every reconnect.
The two seconds are three steps. A sentinel pings the primary continuously and marks it subjectively down only after down-after-milliseconds pass with no answer [7]. A configured quorum then has to agree. Quorum is the guard against a brief network blip, and it is why sentinels run in odd numbers, typically three or five [8]. Then the sentinels elect a leader among themselves, and the leader picks the most up-to-date replica and promotes it with REPLICAOF NO ONE, repointing the other replicas at the new primary [9].
Subtracting the timer setting from the measured recoveries leaves 2.3 to 2.4 seconds of vote, election and promotion [15]. Hold that overhead steady and a down-after of one second predicts about 3.3 seconds of failed writes, with the fixed part accounting for roughly 70 percent of the window [16]. Cutting the timer also leans more of the false-positive protection onto quorum, because the timer is the other filter [7][8]. "Nobody tunes it, and it's the single biggest factor in how long your app stares at a dead Redis," the post's author wrote of down-after [6].
Engine choice moved the number by a tenth of a second [20]. Redis 8, Valkey 8 and Dragonfly were killed under identical sentinels [4]. "The outage belongs to Sentinel, not to the engine," the author wrote [5]. A tenth of a second is not a reason to migrate a datastore. The text prints that comparison at one setting, down-after=5s, and points to a fuller table further down the article [19].
For 7.3 seconds to be your number, your sentinels have to reach agreement as fast as three processes on one Docker host do [2]. Fifteen kills spread over three settings and three engines is about 1.7 runs per combination [17], so the run-to-run spread inside any single combination is not established here. The clock stops server-side, at the first successful write on the promoted replica [2], and the post does not name the client library that issued those writes [18].
Client redirection is the fourth piece. The application asks a sentinel who the master is right now and connects to whatever it is told [10]. An app holding a hardcoded primary address has nowhere new to write when the promotion completes.
The shape under test is a common one: several app instances behind a load balancer, all pointed at a single Redis carrying cache, sessions and the Horizon queue [12]. Sentinel is the recommendation there when the data fits on one node's worth of RAM, with Cluster held back for keyspaces or write rates too big for a single node [14]. The lab, the kill script and the sweep are published at github.com/sunnysahijwani/redis-sentinel-failover-lab, where the instruction is docker compose up and kill the primary yourself [11].
What to watch
- A rerun with the three sentinels in separate failure domains would show whether quorum agreement still fits inside two seconds.
- Reproductions of the sweep outside Docker on a single host: 2.3 seconds of overhead may be an artefact of colocated sentinels.
- Application-layer timing, showing how long Laravel sessions and Horizon jobs actually stall after the promotion completes.