Batching is the mechanism behind the speedup, and it comes with a behavioural cost. The streaming engine cuts a query into pieces instead of keeping the whole dataset and its intermediate results resident, which is what buys execution over data larger than RAM [4]. Nothing in that model carries the input's incidental ordering through a join, a group_by or an unpivot, so the values arrive correct and the sequence does not match what you had [5]. Polars' migration guide gives two answers: sort explicitly, or ask for observable order with settings such as `maintain_order="left"` [6]. There is also a one-line retreat, `pl.Config.set_engine_affinity("in-memory")`, which pins the old engine globally, and the same choice can be made per query [7].
On speed, Polars says the streaming engine should be about five times faster in aggregate, and is clear that the figure comes from its own benchmarks rather than independent testing [8]. Set that against the published per-scale numbers and the aggregate claim is exactly twice the small-scale result: 5 divided by 2.5 [16]. So the headline multiple sits near the SF100 end of the range, not the SF10 end [9]. For it to transfer you need a query mix that resembles PDS-H, data volumes closer to SF100 than SF10, and comparable hardware. Polars itself notes that PDS-H is derived from TPC-H without being audited or compliant, and that results vary by query and by machine [10].
The strictness changes are easier to plan for, because they fail loudly. Horizontal concatenation of unequal-height frames now raises `ShapeError` instead of padding the shorter one with nulls, and `horizontal_extend` is there if padding was the intent [13]. Ambiguous string-to-date casts are gone in favour of dedicated parsers such as `str.to_date()`, and removed methods and arguments increasingly raise `AttributeRemovedError` or `ArgumentRemovedError` with a pointer to the replacement [14]. The `is_in` case is the one worth reading twice: exact integer representation in a 64-bit float runs out at 9,007,199,254,740,992 [17], and above that a user ID could round into its neighbour and match the wrong person [12]. Two customers quietly becoming one customer is a memorable way to lose a row.
That asymmetry is the whole migration plan. Type and shape changes announce themselves with an error and a line number pointing at the fix. An ordering assumption that nobody wrote down leaves nothing to grep for.
This fits the direction Vink has been pushing all along. Vink tried copying pandas' API early on and said in a 2024 interview that it clashed with his preference for predictable output types and stricter contracts [20], and the major version exists precisely to remove old design decisions and change defaults that were constraining the query engine [18]. In my context, where batch outputs get sorted before they are written anyway, the explicit sort is cheap and the engine pin is not: pinning keeps you on the path upstream is walking away from. If you cannot afford the sort today, pin it, but treat that as a dated stopgap rather than a setting.
