Build1 publisher2 min readPublished
SeaTunnel drops data from new Kafka partitions until you enable discovery
Apache SeaTunnel's Kafka source scans a topic's partitions once, when the job starts. Any partition added afterward goes unread and its records are lost until you set partition-discovery.interval-millis to a positive value.
The Engineer · Build desk
What happened
- By default, expanding a Kafka topic from two to four partitions leaves a running SeaTunnel job consuming neither new partition, and their records go unprocessed until the job restarts.
- Restarting to pick up new partitions can, with start_mode EARLIEST, reread reassigned partitions from the beginning and produce duplicate records.
- With the parameter set, the demo job absorbed the expansion live: all six post-expansion messages, including those on the new partitions, reached the console with no restart.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure The routine act of adding partitions to a busy topic is what triggers the loss, so any team scaling a Kafka source for capacity is exposed by default.
- decision Discovery is off unless configured, so the safe move is to set partition-discovery.interval-millis when the job is written, not after a topic has already grown.
- cost Choosing the restart fallback costs a full job pause, and if you rely on checkpoints there is a data-loss risk on recovery, so the recovery path can itself drop records.
SeaTunnel's Kafka source enumerates a topic's partitions when the job starts, and without extra configuration it never looks for the ones added afterward [2]. Kafka allows the change anyway: you can add partitions to a live topic, and the producer immediately spreads records across all of them, including the new ones [10]. The job stays up. It simply stops receiving whatever the broker now writes to the partitions it never assigned [1].
The SeaTunnel walkthrough reproduces the failure in a handful of commands. A topic called ksource starts with two partitions, and three messages land on partitions 0 and 1 and show up in the console [9]. Expand the topic to four, produce six more messages, and Kafka's default partitioning spreads them across all four, with some landing on the new partitions 2 and 3 [10]. Take the discovery parameter back out and repeat, and the records on partitions 2 and 3 are never consumed, while everything produced before the expansion still flows through [12].
Set to a positive value, partition-discovery.interval-millis makes SeaTunnel rescan Kafka on an interval and attach any new partitions to the running consumer [3]. The walkthrough uses 5000, so the scan runs every five seconds [4].
SeaTunnel's other option is a restart. The whole job pauses while it comes back, and if you rely on checkpoints for recovery there is a risk of missing data depending on the offset state at the time [5]. With start_mode set to EARLIEST, a restart can also reread the reassigned partitions from the beginning, replaying old data as duplicates [6]. Interval discovery avoids that replay, because EARLIEST on a freshly discovered partition reads from its earliest offset, which for a brand-new partition is the start of an empty log [7].
On Apache SeaTunnel 2.3.12, a running job survives a topic scale-up cleanly only if someone set partition-discovery.interval-millis before the scale-up happened [8][1]. By default, that ordinary capacity move quietly drops records from a job that keeps running [1][12].
What to watch
- Whether a later SeaTunnel release changes the default so partition discovery is on, or logs a warning when a consumed topic gains partitions.
- Whether the same startup-only scan applies to SeaTunnel's other message-queue sources, or is specific to the Kafka source.
- Whether interval discovery handles partitions removed or a topic replaced, not only partitions added, which the walkthrough does not cover.