Build1 publisher2 min readPublished
Databricks rewrites the state store on restart so a stateful stream can change partition count
The partition count was frozen at checkpoint creation because each partition has its own RocksDB instance on disk. In Databricks Runtime 18, a separate config resizes the store and moves the state to match.
The Engineer · Build desk
What happened
- Databricks has put on-demand state repartitioning into Public Preview on Databricks Runtime 18 and above, letting a stateful Structured Streaming query be resized while its checkpoint state stays intact.
- Until now the partition count was baked into the checkpoint when the stream first started, so raising spark.sql.shuffle.partitions and restarting a stateful query changed nothing.
- The only way to apply a new number was to abandon the checkpoint and start over, which for a stateful query meant discarding all the accumulated state.
- Databricks says the feature covers any stateful query, including aggregations, stream-stream joins, deduplication, sessionization and transformWithState.
- Alexis Chicoine of Coveo, an early adopter, said the capability cut the company's related Amazon S3 API costs by 40 percent.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision The partition count picked on day one is now revisable for anyone on DBR 18 with the default state store, so sizing becomes a maintenance-window decision instead of a permanent one.
- constraint The resize still costs a query stop and a one-time redistribution pass before normal processing resumes, so it has to be scheduled rather than triggered by a traffic spike.
- cost The saving Coveo reports came off its storage API line, so teams whose object-store request charges are already small next to compute should expect the bill to move less.
- capability State layout now has its own setting, so tuning shuffle width for current traffic no longer implies changing the on-disk partitioning of the state store.
The freeze was defensive. State is partitioned by a hash of the key, each partition's data lives in its own RocksDB instance inside the checkpoint, and the partition count therefore defines the layout of the entire state store on disk [5]. Change the count and the hash lands somewhere else. Databricks' example is a key that used to live in partition 47 and now hashes to partition 12, while its accumulated state is still sitting in the original partition's files [6]. The query would, Databricks wrote, "lose track of its own memory", so Structured Streaming locked the count at checkpoint creation and ignored later changes to `spark.sql.shuffle.partitions` [7]. That was the correct call and an inconvenient one.
What makes the new value stick is that it is a different setting. For stateful queries, `spark.sql.streaming.stateStore.partitions` takes precedence over `spark.sql.shuffle.partitions` [8]. The sequence is the one every streaming developer already runs: stop, reconfigure, restart. On restart the query does not resume normal processing immediately. It finishes the last planned microbatch if one is pending, then physically redistributes the state to match the new partition count [9].
The size of that redistribution is set by the number you accepted at the start. The default is 200 shuffle partitions [4], which is 200 RocksDB instances inside one checkpoint, one per partition [14]. Going to 400 is a rewrite of that on-disk layout, not a metadata update [5][9].
Coveo's saving showed up in its S3 bill. Alexis Chicoine, Senior Software Developer at Coveo, said the old choice was the expensive part: "Before, every scaling decision forced a trade-off: either overprovision or rebuild from new checkpoints, which drove storage API costs nearly the same as compute costs." [13] Chicoine also said the company now scales "without disrupting the existing state or triggering costly checkpoint migrations" [15]. For a reduction on that order to appear in another pipeline, checkpoint rebuilds and overprovisioned partition counts would have to be driving a comparable share of its object-store request volume [12]. Where storage API charges already sit well below compute, the same change reduces a smaller number.
What to watch
- Whether on-demand state repartitioning leaves Public Preview, and whether the requirement for the default state store is relaxed for other state store providers.
- Published timings for the one-time redistribution pass on a large state store, so teams can size the restart window.
- Whether the capability appears in open-source Apache Spark or stays specific to Databricks Runtime.