A Spark benchmark's Arrow UDF row toggled a setting that never touches UDFs

The failure is a naming problem. Every Python UDF call makes three separate choices: how data crosses from the JVM to the Python worker, which physical operator Spark plans (BatchEvalPython or ArrowEvalPython), and what the function receives once the data lands ^[8]. One config key sounds like it covers all three. According to the post's author, that key never touches Python UDFs ^[4], and the PySpark documentation says it controls pandas conversions ^[6].

That makes the original table an A/A test. The two rows were the same pickled UDF measured twice ^[7]. The gap was 0.8% at 50M rows and 0.3% at 100M ^[14]. The author's own phrase for it: "I had built an A/A test by accident" ^[13]. The post documents the author's write-up leaning on this pair as its key insight ^[3].

With the right switch, the benchmark's own numbers move. useArrow=True took arithmetic from 5.20s to 2.23s and the CDF workload from 6.18s to 2.94s ^[9]. That is 2.3x and 2.1x ^[15]. The original line, that Arrow buys nothing when the function runs one row at a time ^[3], is wrong as written.

The author's replacement thesis is that a UDF's cost depends far more on what the function receives than on whether Arrow carries the bytes ^[12]. The data fits that ordering. Vectorized UDFs cross into Python once per batch, and on arithmetic they ran about 3x faster again than useArrow=True ^[11]. Transport is worth about 2x. Execution model is worth roughly another 3x.

One figure in the post does not sit well with its own table. The author gives a ceiling of roughly 8 to 10 million rows per second for row-at-a-time UDFs, citing 50M rows in 5.2s and 100M in 10.1s ^[10]. Those work out to 9.6 and 9.9 million rows per second ^[16]. That fits the pickle UDF. If the 2.23s Arrow time comes from the same 50M-row run as the 5.20s baseline, useArrow=True ran at about 22 million rows per second ^[17]. Until the post gives the scale of the 2.23s figure, we would not quote the ceiling for Arrow-backed per-row UDFs.

Transfer is the other limit. These are one author's runs of x * 2 + 1, a string transform and a normal CDF ^[1]. The excerpt does not include the Spark version or the hardware. The 2x is serialization cost saved per row. In our view it carries to your job only if the function body is cheap enough that transport is a large share of the runtime. A UDF that spends its time in a model call will see less.

The check is short. Search job configs and benchmark harnesses for spark.sql.execution.arrow.pyspark.enabled. The settings that move a regular UDF are spark.sql.execution.pythonUDF.arrow.enabled for the session and useArrow=True per UDF ^[5]. Then look at the physical plan, since the two transports plan different operators ^[8].