BuildNot yet confirmed elsewhere1 publisher2 min readPublished
Polars 2.0's streaming default changes left-join row order from run to run
Polars 2.0.0 runs lazy queries on the streaming engine by default, and a left join on that engine no longer keeps row order. A dev.to author's repro shows the change raising no error and leaving every value correct.
The Engineer · Build desk
The repro left-joins 2,000,000 order rows to 100,000 customer rows. Under Polars 1.44.2 all three runs began at order_id 0. Under 2.0.0 run 0 began at order_id 500015, and runs 1 and 2 at 625013 and 375038.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Order rows left-joined in the repro | 2,000,000 rows | 4 | |
| First order_id on 1.44.2 | 0 all three runs | 5 | |
| First order_id on 2.0.0, run 0 | 500,015 order_id | runs 1 and 2: 625013 and 375038 | 6 |
What happened
- Polars 2.0.0 shipped on 6 October, and most of its breaking changes are loud: a removed argument raises and names its replacement.
- A dev.to author ran five self-contained snippets under 1.44.2 and 2.0.0 and counts three silent changes among the five.
- On 1.44.2 the 2-million-row join was in order on all three runs; on 2.0.0 it was out of order on all three, and the first order_id differed each run (500015, 625013, 375038).
- Adding an Int64 to a UInt64 now returns an exact Int128, where 1.44.2 rounded through Float64.
- Exploding a list column no longer emits a null row for an empty list, so row counts drop unless you pass empty_as_null=True.
Why it matters
- exposure Row numbers, zipped lists and positional files read by the next job are the exposed consumers: values stay correct, positions move, and nothing raises.
- constraint A green suite on small fixtures does not clear the upgrade. The check has to run the join at scale and more than once, because positions moved differently on every run.
- cost Downstream systems without 128-bit integers, such as a database column or another tool's Parquet reader, absorb the Int128 change. Casting to Float64 restores the old type, and that type was off by 1 at this size.
- decision Choosing horizontal_extend to keep the old padding means deciding whether to hide a failed upstream job. In the post's example the padding did.
The cause is a stated ordering rule. According to the post, the streaming engine does not promise row order for operations that do not need it, and a left join is one of them [3]. The repro joins 2,000,000 orders to 100,000 customers on customer_id and passes no maintain_order argument [4]. Under 1.44.2 that code returned sorted output anyway on every run [5].
The fix is one keyword. Passing join(..., maintain_order="left") returned rows in order on all three runs under 2.0.0 [17]. The shortcut is collect(engine="in-memory") on its own. The author says it kept order in the small repro, but a scan-based join in the book came back unordered under it [18].
Treat the repro as a claim about one workload: synthetic integer keys, three runs, 2 million rows. The post does not describe the hardware, give a size at which reordering starts, or cite Polars release notes. It says only that small test fixtures often keep their order [16]. In our view the change reaches a pipeline only when something downstream reads row position and nothing sorts after the join. A job that re-sorts on a key, or looks rows up by key, would not see it.
The other changes differ in how long they stay hidden. The integer add now returns Int128 and is exact. On 1.44.2 it returned Float64 18455751272964292608 against an exact 18455751272964292607, an error of 1 [8][14].
For explode, the post's three-row input gave 4 rows on 1.44.2 and 3 on 2.0.0, a difference of 1. Calling explode("tags", empty_as_null=True) restores the old rows [10][15].
Horizontal concat raises on unequal heights, but only on a day the heights differ, so it can pass an upgrade and fail in production a week later [11]. The is_in check of an Int64 value against a List(Float64) raises too, and the error tells you to cast one side [12]. The side matters. Casting the float list to Int64 truncates 1.99 to 1 and returns True, while casting the needle to Float64 returns False [12].
What to watch
- Whether Polars documentation or release notes list which other lazy operations lose order under the streaming default; the post names only left join as one of them.
- Whether anyone reproduces the scan-based join that came back unordered under collect(engine="in-memory"); the book case is not published in the post.
- Whether other tools' Parquet readers and database drivers accept Int128 output, or whether teams end up casting back to Float64.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence62
- Adoption
- Insufficient
- Hype gap+8
- Incentives40
- Confidence60
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Polars 2.0.0 shipped on 6 October. Most of its breaking changes are loud: a removed argument raises and tells you what to use instead.
ReportedSupportedSource: dev.to post 'Polars 2.0: five things that change when you upgrade, three of them silently'View cited source - [2]
Lazy queries in Polars 2.0.0 now run on the streaming engine by default.
- [3]
The streaming engine does not promise row order for operations that don't need it, and a left join is one of them.
- [4]
The repro left-joins a 2,000,000-row orders LazyFrame to a 100,000-row customers LazyFrame on customer_id, collects it three times, and passes no maintain_order argument.
- [5]
Under 1.44.2 all three runs of the join printed in_order=True with first five order_ids [0, 1, 2, 3, 4].
- [6]
Under 2.0.0 all three runs printed in_order=False, with first five order_ids starting at 500015, 625013 and 375038 on runs 0, 1 and 2.
- [7]
Every value is still correct; only the row positions move, and they move differently on every run. Anything downstream that relies on position (row numbers, zipping with another list, a positional file the next job reads) is now wrong without an error.
- [8]
Adding an Int64 column value 9007199254740993 to a UInt64 value 18446744073709551614: 1.44.2 returned Float64 18455751272964292608, 2.0.0 returns Int128 18455751272964292607, and the exact sum is 18455751272964292607.
- [9]
The dtype change means anything downstream without 128-bit integers (a database column, Parquet readers in other tools) will notice; adding .cast(pl.Float64) gives the old type.
- [10]
Exploding a three-row list column containing an empty list gave 4 rows ['a', 'b', None, 'c'] on 1.44.2 (with a DeprecationWarning) and 3 rows ['a', 'b', 'c'] on 2.0.0; explode("tags", empty_as_null=True) restores the old rows on 2.0.0.
- [11]
Horizontal concat of frames with different heights padded with null on 1.44.2 and raises ShapeError on 2.0.0; it raises only on the day the heights differ, so it can pass an upgrade and fail in production a week later. how="horizontal_extend" keeps the padding, but in the example the padding was hiding a failed upstream job.
- [12]
is_in of an Int64 value against a List(Float64) printed [False] on 1.44.2 and raises InvalidOperationError on 2.0.0, with an error saying to cast one side. Casting the float list to Int64 truncates 1.99 to 1 and returns [True]; casting the needle to Float64 returns [False].
- [13]
The post's headline says five things change on upgrade, three of them silently; the author ran five self-contained snippets under 1.44.2 and 2.0.0.
- [14]
The 1.44.2 Float64 result was off by 1 from the exact sum.
- [15]
The explode output has 1 fewer row on 2.0.0 than on 1.44.2 for the post's input.
- [16]
Small test fixtures often keep their order, which is why a green test suite proves little here.
ReportedContestedSource: dev.to post on Polars 2.0 upgrade changes2 sources— create a free account to open themView cited source - [17]
join(..., maintain_order="left") came back in order on all three runs under 2.0.0.
ReportedContestedSource: dev.to post on Polars 2.0 upgrade changes2 sources— create a free account to open themView cited source - [18]
collect(engine="in-memory") alone kept order in the small repro, but a scan-based join in the author's book came back unordered under it.
ReportedContestedSource: dev.to post on Polars 2.0 upgrade changes2 sources— create a free account to open themView cited source
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toPolars 2.0: five things that change when you upgrade, three of them silently
1 article · October 12, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- Breaking Changes and MigrationFollow
- Data pipeline correctnessFollow
- DataFrame librariesFollow