PySpark's Arrow paths became defaults in Spark 4.2 after a ticket filed in 2016
Spark 4.2 makes the Arrow paths PySpark's defaults, according to a dev.to history of how the format reached each JVM-to-Python boundary. Teams upgrading get that serialization without opting in, so jobs with UDFs and pandas conversions need a test run first.
The Engineer · Build desk
Key figures from a dev.to post: three orders of magnitude of overhead wrapped around a UDF like return x * 1.1; four serialization steps paid by every row; and eight useful int64 values in a 64-byte cache line when columns are stored contiguously, instead of one row's mixed fields.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Orders of magnitude of overhead around a UDF like x * 1.1 | 3 orders | 18 | |
| Serialization steps per row | 4 steps | 11 | |
| Useful int64 values per 64-byte cache line | 8 int64 values | with columns stored contiguously | 14 |
What happened
- Wes McKinney filed SPARK-13534 on February 28, 2016, asking for one method, toPandas(), to stop using pickle. It took two years to ship.
- The same idea then went through UDFs, partition-level APIs, the Spark Connect network protocol and custom data sources.
- Spark 4.1 let Python functions take Spark's data as Arrow arrays, with pandas left out entirely.
- According to the post, the conversion that remains now lives inside Spark's own engine.
Why it matters
- decision Teams moving to 4.2 get Arrow paths they did not turn on, so every place their code pulls data out of the JVM becomes a behavior change to test as well as a speedup.
- constraint The overhead case rests on a one-line UDF. Jobs whose functions do real work per row spend a smaller share of runtime on serialization and will see less.
- exposure The 4.2 default flip is stated in one sentence of a single blog post that does not list the settings involved, so a migration plan built on it alone needs the release notes to confirm which paths changed.
Before Arrow, every row crossed on its own. For most of Spark's life pickle was the only route, and it carried a single row per trip [10]. Each row paid four serialization steps and a function call [11]. For a UDF like `return x * 1.1`, the post says, the arithmetic takes nanoseconds and the machinery around it takes microseconds, three orders of magnitude of overhead around almost no work [18]. The multiplication was the cheap part.
Arrow's idea is to agree on the bytes. It specifies a language-independent, columnar, in-memory layout down to byte alignment, null bitmaps and buffer order, so any two systems that understand Arrow can pass data between them with no conversion step [12]. Inside a single process, or in shared memory, the hand-off is zero-copy. Across a socket it is still a copy, but neither side decodes row by row, because the bytes on the wire are already the bytes in memory [13]. Columnar storage helps the CPU as well: a 64-byte cache line fills with eight int64 values that are all useful, where row storage would fill it with one row's mixed fields [14]. The post leans on the MonetDB/X100 paper (CIDR 2005), which argued that pushing batches of around a thousand column values through each operator is an order of magnitude or two faster than going one row at a time [15].
The fixes came one door at a time because PySpark is a Python client for a JVM engine. Python builds the query plan and the JVM executes it, so DataFrame code runs at the same speed in PySpark and Scala [8]. Data crosses only when your own code needs it [9]. McKinney's ticket, titled "Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas," covered one such crossing [3]. He filed it in the same month Arrow was announced as an Apache top-level project [2][16].
The author's framing is that each release removed one conversion at one boundary, and that what each left behind explains the next [1]. On that account, Arrow arrays handed straight to Python functions in Spark 4.1 and the 4.2 defaults arrived in adjacent minor releases [5][6][17].
In our view the upgrade work is an inventory of your own crossings: every UDF, every `toPandas()` call, every `createDataFrame(pandas_df)` call and every custom data source [9].
What to watch
- The Spark 4.2 release notes and migration guide, which would show exactly which settings and paths the default change covers.
- Measurements of the 4.2 defaults on UDFs that do real work per row, since the post's overhead figure comes from a one-line function.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence55
- Adoption
- Insufficient
- Hype gap+10
- Incentives
- Insufficient
- Confidence58
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Arrow's contribution to PySpark is a shared memory format at every boundary between the JVM and Python; each release removed one conversion at one boundary, and what each one left behind explains the next.
- [2]
Wes McKinney filed SPARK-13534 on February 28, 2016, asking for one method, DataFrame.toPandas, to stop using pickle; it took two years to ship.
- [3]
The ticket was titled "Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas."
- [4]
Over the decade after the ticket, the Arrow idea went through UDFs, partition-level APIs, the network protocol of Spark Connect and custom data sources.
- [5]
In Spark 4.1 a Python function could receive Spark's data as an Arrow array and never touch pandas at all.
- [6]
In Spark 4.2 the Arrow paths became the defaults.
- [7]
The conversion that remains now lives inside Spark's own engine rather than at the Python edge.
- [8]
PySpark is a Python client for a JVM engine: Python builds a query plan and hands it to the JVM, which optimizes, distributes and executes it, so DataFrame code runs at the same speed in PySpark and Scala.
- [9]
Data has to physically cross from the JVM into a Python process when your own code needs it: a UDF, toPandas(), createDataFrame(pandas_df) or a custom data source.
- [10]
For most of Spark's history the only way across the JVM-Python boundary was pickle, one row at a time.
- [11]
Every row pays four serialization steps and a function call.
- [12]
Arrow is a language-independent, columnar, in-memory layout specified down to byte alignment, null bitmaps and buffer order, so two Arrow-aware systems can hand each other data without converting it.
- [13]
Within a process or with shared memory the hand-off is zero-copy; across a socket it is still a copy, but with no per-row decoding on either side because the bytes on the wire are already the bytes in memory.
- [14]
With columns stored contiguously, a 64-byte cache line holds eight useful int64 values instead of one row's mixed fields.
- [15]
The MonetDB/X100 work (CIDR 2005) argued that pushing batches of around a thousand column values through each operator, instead of one row at a time, is an order of magnitude or two faster.
- [16]
Arrow was announced as an Apache top-level project in February 2016, skipping the incubator.
- [17]
The Arrow-array input for Python functions (Spark 4.1) and the Arrow defaults (Spark 4.2) arrived one minor release apart.
- [18]
For a UDF like `return x * 1.1`, the arithmetic is nanoseconds and the machinery around it is microseconds, three orders of magnitude of overhead wrapped around almost no work.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toExplaining the Python-JVM Boundary in Spark, and What Arrow Changed
1 article · October 13, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Entities
- pandasFollow
- Apache SparkFollow
- PySparkFollow
- Spark ConnectFollow
- Apache ArrowFollow
- Wes McKinneyFollow