Data pipeline cost optimisation

Databricks rewrites the state store on restart so a stateful stream can change partition count

The partition count was frozen at checkpoint creation because each partition has its own RocksDB instance on disk. In Databricks Runtime 18, a separate config resizes the store and moves the state to match.