Product1 publisher2 min readPublished
Vitess v25 makes emergency failover wait only on the tablets that could win the election
Before v25, one surviving tablet that missed the relay-log deadline could fail an entire EmergencyReparentShard. Vitess says the new release filters that wait down to tablets sharing the leading history.
The Product Desk · Product desk
What happened
- Vitess runs EmergencyReparentShard when a shard's primary is dead or unreachable, picking a replacement from only the surviving tablets, while PlannedReparentShard takes a clean handoff from a healthy primary.
- Before v25, the relay-log apply phase waited on every surviving tablet still under consideration, and a single tablet exceeding --wait-replicas-timeout failed the whole operation.
- Vitess says v25 filters that phase by received GTIDs and races relay-log apply across the tablets sharing the leading history, for faster and less brittle emergency reparents.
- The project says the problem has been present since ERS was introduced and was first formally reported as issue #18529 around the Vitess 22 release.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- capability An emergency reparent that used to abort because one lagging tablet outside the promotion race ran past the timeout can now complete, so the time before writes resume depends less on the slowest survivor.
- decision Any team that padded --wait-replicas-timeout above the 15 or 30 second defaults to get failovers through now has to decide whether that number is still right when the wait covers fewer tablets.
- cost Under the old behaviour, the shards with the most replicas carried the most waits that could expire, so the more read replicas a shard had, the less reliable its failover.
A replica can hold a transaction in its relay log and still be some way from having applied it. ERS therefore compares two MySQL positions for every candidate, what it received and what it applied, and applied position only entered candidate sorting in v23 [13][14]. Before promoting anything, ERS waits for the candidate to finish applying what it already received [13].
Version 25 keeps that wait and narrows who is in it. The phase is filtered by received GTIDs, and relay-log apply is raced across the tablets that share the leading history [9]. The Vitess post puts the principle this way: "Waiting for the eventual primary is necessary; letting a tablet that cannot lead the election fail the entire operation is not." [12]
The post's own example shard has four tablets: an unavailable primary, two replicas and one rdonly [19]. With the primary gone, three tablets survive, so the old phase held three separate waits, any one of which could reach the 15-second default in vtctldclient and end the reparent [20]. Under VTOrc the budget is 30 seconds, twice as long [21]. That second number is the one most teams live with, because VTOrc is what resolves many unplanned failures automatically [4].
The selection goal is the same as before: promote the most-advanced surviving history quickly, because the shard takes no writes until it does [5]. The most-advanced tablet still may not be the one promoted. Promotion rules, cell restrictions and durability requirements can select a different tablet, which then has to catch up from that source, and any required semi-sync acknowledgers have to be ready before promotion completes [15][16]. The post describes the change qualitatively and does not publish before-and-after failover timings [23].
For a team working out what to re-test before upgrading, two things sort the shards: how many tablets ERS would consider, and whether any of them habitually trails on applied position while keeping up on received. Shards high on both are where pre-v25 ERS timed out and where the GTID filter changes the result [7][11]. A three-tablet shard whose replicas keep up will behave about as it did.
If a runbook raised --wait-replicas-timeout to get emergency reparents to complete, that number was extending the wait for tablets that could not lead the election, and after the upgrade it covers a smaller set [8][9]. A team that wants to see the difference can kill the primary in staging on a shard with one deliberately lagging replica; ERS on the current version and ERS on v25 will diverge there.
What to watch
- Whether the v25 release notes carry before-and-after ERS timings.
- Whether issue #18529 closes with v25 or stays open for the remaining wait on the eventual primary.
- Whether the GTID filtering ships as default behaviour in both vtctldclient and VTOrc or behind a flag.