A dev.to author sorts Spark's modules by whether a thin Python client can reach them
A dev.to author measured Spark 4.2.0's Python payload at roughly 480 MB for full PySpark and about 20 MB for pyspark-client. The post argues that modules unreachable from that client, such as GraphX, SparkR and DStreams, are the ones fading.
The Engineer · Build desk
Three figures. A working thin-client virtualenv came to at least 360 MB, roughly 360 to 380 MB across the author's installs. The pyspark-client wheel is 1.60 MB. Spark's own Python payload in pyspark-client is about 20 MB, in the author's measurement of the 4.2.0 packages.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Working thin-client virtualenv, at least | 360 MB | roughly 360 to 380 MB across installs | 11 |
| pyspark-client wheel | 1.6 MB | 11 | |
| Spark's own Python payload in pyspark-client, about | 20 MB | author's measurement, 4.2.0 packages | 10 |
What happened
- A PySpark 4.2 Connect session rejects _jvm by name, and it rejects _jsc, sparkContext and the other Py4J attributes with it.
- The spark.api.mode setting still defaults to classic, and the proposal to make Connect the default, SPARK-50411, is open.
- The pyspark-client wheel is 1.60 MB, but a working thin-client virtualenv came to roughly 360 to 380 MB because pyarrow, pandas and numpy are hard requirements.
Why it matters
- constraint Modules built on RDDs cannot cross to a thin client, because an RDD is a handle to partitioned data and closures in the driver's JVM and not a plan.
- decision Any helper that reaches into _jvm or sparkContext ties its job to classic mode, so teams must rewrite it against the DataFrame API now or carry it until the default changes.
- exposure Python real-time aggregation and deduplication exist only on the 4.3 branch, and the vote on the first 4.3 release candidate failed on September 6.
The sorting follows from what a Connect request is. The client builds an unresolved logical plan, serializes it as protobuf and sends it over gRPC. The server plans and runs it, and results come back as Arrow batches [1]. Anything expressible as a plan travels. Code that assumes the client and the driver share a JVM stays behind [2]. Spark SQL passes almost by definition, since it is the machinery that turns logical plans into execution [3].
The author's own case is the cleanest evidence in the post. In February, testing Real-Time Mode on open-source Spark 4.1, the author found the real-time trigger in Scala and could not ask for it from PySpark. The workaround was a helper that went in through the Py4J gateway and constructed the Scala trigger object itself [4]. It worked only in classic mode, because a Connect client has no JVM to reach into [5]. The author's verdict was that it is "exactly the kind of code that has no future" [15]. Spark 4.2.0, released July 14, 2026, added trigger(realTime=...) to PySpark [6][7].
Full PySpark carries roughly 277 jars, and pyspark-client leaves every one of them out. Measuring the 4.2.0 packages, the author found Spark's own share of the Python install falling from roughly 480 MB to about 20 MB [10]. That is a drop of about 96 percent [16].
Spark's 20 MB is also only about 5 to 6 percent of the 360 to 380 MB virtualenvs the author ended up with [11][17]. The post does not give a comparable virtualenv size for full PySpark, so the saving across a whole environment goes unmeasured. The author put it this way: "The honest headline is no JVM, no JDK, no jars, not small." [14]
The "fading" label is the author's inference from what cannot cross [18]. A module that cannot cross can still be maintained in classic mode, and classic remains the default [12]. In our view the filter is sound for any code that needs a driver JVM, because a plan cannot carry it.
For new code the author is blunt. Code that touches sparkContext or _jvm "is a bet against where the project is going" [13].
What to watch
- Whether SPARK-50411 closes and spark.api.mode stops defaulting to classic.
- Whether a 4.3 release candidate passes its vote with the Python real-time aggregation and deduplication path intact.
- Whether the rebuilt MLlib and Structured Streaming paths the post describes land as Connect-reachable APIs.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence58
- Adoption
- Insufficient
- Hype gap+8
- Incentives
- Insufficient
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Spark Connect splits application from engine: the client builds an unresolved logical plan, serializes it as protobuf, sends it over gRPC, and gets results back as Arrow batches, while everything that plans and runs the query lives on the server.
- [2]
Anything expressible as a logical plan travels over Spark Connect; anything that assumes the client and the driver share a JVM does not.
- [3]
Spark SQL passes the thin-client test almost by definition, because a Connect request is a logical plan and Spark SQL is the machinery that turns logical plans into execution.
- [4]
In February the author tested Real-Time Mode on open-source Spark 4.1; the real-time trigger existed in Scala but PySpark had no way to ask for it, so the author wrote a helper that reached through the Py4J gateway and built the Scala trigger object by hand.
- [5]
The Py4J helper only runs in classic mode, because a Spark Connect client has no JVM to reach into; the PySpark 4.2 Connect session refuses _jvm by name, along with _jsc, sparkContext and the rest of the Py4J surface.
- [6]
Spark 4.2 added trigger(realTime=...) to PySpark, so the same real-time pipeline runs from a thin client.
- [7]
Spark 4.2.0 was released July 14, 2026; the first release candidate of Spark 4.3 was tagged on August 31 and the vote on it failed on September 6.
- [8]
The Spark 4.3 branch lets Python code run aggregations and deduplication in real time too; the author describes what is on the release branch, not what has shipped.
- [9]
An RDD is not a plan but a handle to partitioned data and closures living in the driver's JVM, and every module built on RDDs inherits that limitation.
- [10]
pyspark-client ships only the gRPC client, the protobuf definitions and the DataFrame API, installing none of the roughly 277 jars full pyspark carries; in the author's measurement of the 4.2.0 packages, Spark's own Python payload dropped from roughly 480 MB to about 20 MB and the jar directory to zero.
- [11]
The 1.60 MB figure is the wheel; a working thin-client virtualenv came to roughly 360 to 380 MB across the author's installs, because pyarrow, pandas and numpy are hard requirements.
- [12]
The classic/Connect choice is controlled by spark.api.mode, whose default is still classic; the proposal to make Connect the default (SPARK-50411) is open.
- [13]
The author wrote that anything you write that touches sparkContext or _jvm "is a bet against where the project is going."
- [14]
The author wrote: "The honest headline is no JVM, no JDK, no jars, not small."
- [15]
The author said of the Py4J helper: "it's exactly the kind of code that has no future."
- [16]
The Python payload fell by about 96 percent, from roughly 480 MB to about 20 MB.
- [17]
Spark's own 20 MB is about 5 to 6 percent of a 360 to 380 MB thin-client virtualenv, assuming the 20 MB payload is part of that environment.
- [18]
The author's test is whether a module can be reached from a thin Python client over Spark Connect: SQL and Declarative Pipelines were built for that world, Structured Streaming, MLlib and the Python UDF path are being rebuilt to reach it, and GraphX, SparkR, DStreams and the RDD-based spark.mllib are fading because they cannot cross it.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toSpark's Modules, Seen Through Spark Connect
1 article · October 13, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.