The Apache Software Foundation has released Apache Spark 4.0, the most significant upgrade to the world most widely used big data processing framework since its initial release in 2014. The new version features a completely rewritten query engine called Photon 2.0, native Python execution without JVM overhead, built-in AI inference capabilities, and dramatically improved performance that makes it 3-5x faster than Spark 3.x for typical analytical workloads.
The Photon 2.0 Query Engine
The centerpiece of Spark 4.0 is Photon 2.0, a completely rewritten vectorized query execution engine written in C++ that replaces the Java-based execution engine used in previous versions. Vectorized execution processes data in batches of thousands of rows simultaneously rather than one row at a time, taking advantage of modern CPU SIMD (Single Instruction, Multiple Data) instructions to dramatically accelerate analytical queries.
In benchmarks on the TPC-DS decision support benchmark, Photon 2.0 delivers 3.2x better performance than Spark 3.5 for complex analytical queries involving multiple joins, aggregations, and window functions. For simple aggregation queries on large datasets, the improvement reaches 5x. These performance gains translate directly into lower infrastructure costs — organizations can process the same workloads on fewer machines, or process significantly more data on the same infrastructure.
Photon 2.0 also introduces adaptive query execution improvements that automatically optimize query plans at runtime based on actual data statistics rather than estimates. This is particularly valuable for queries on skewed data — where some partitions are much larger than others — which previously required manual tuning to avoid performance bottlenecks.
Native Python Execution
One of the most significant architectural changes in Spark 4.0 is the introduction of native Python execution through a new component called PyArrow Flight. In previous versions of Spark, Python code executed in PySpark had to serialize data to the JVM, execute the Python function, and deserialize the results back — a process that added significant overhead for Python UDFs (User Defined Functions). PyArrow Flight eliminates this overhead by executing Python code natively without JVM involvement, using Apache Arrow as the in-memory data format for zero-copy data sharing between the Python and Spark execution environments.
The performance improvement for Python-heavy workloads is dramatic. Python UDFs that previously ran 10-20x slower than equivalent Scala or Java code now run at near-native speed. This change is particularly significant for data science workloads that use Python libraries like NumPy, Pandas, and scikit-learn within Spark pipelines, as these libraries can now be used without the performance penalty that previously made them impractical for large-scale processing.
AI-Native Features
Spark 4.0 introduces built-in support for AI inference workloads through a new MLflow integration that allows trained machine learning models to be deployed as Spark UDFs with a single line of code. This makes it trivial to apply a trained model to a large dataset in a distributed fashion — a task that previously required significant custom engineering. The integration supports models trained in any framework including TensorFlow, PyTorch, scikit-learn, and XGBoost.
A new AI Functions library provides SQL-accessible functions for common AI tasks including text classification, sentiment analysis, entity extraction, and embedding generation. These functions can be called directly in Spark SQL queries, making AI capabilities accessible to data analysts who are comfortable with SQL but not Python or machine learning frameworks. The functions are backed by configurable model endpoints, allowing organizations to use their own fine-tuned models or commercial API providers.
Streaming Improvements
Spark Structured Streaming has been significantly enhanced in version 4.0 with the introduction of Stateful Processing 2.0, which provides more flexible and efficient management of streaming state. The new state management system supports arbitrary state schemas, time-to-live (TTL) for state cleanup, and state migration for schema evolution — capabilities that were previously difficult or impossible to implement in Spark Streaming.
Latency for streaming workloads has been reduced by 40% through improvements to the micro-batch scheduling algorithm and the introduction of a new continuous processing mode that eliminates micro-batch overhead for latency-sensitive applications. Organizations running real-time fraud detection, live recommendation systems, and IoT data processing pipelines will see immediate benefits from these improvements.
Migration from Spark 3.x
The Spark team has prioritized backward compatibility in version 4.0, and most Spark 3.x applications can be migrated with minimal changes. The migration guide identifies a small number of breaking changes, primarily related to the removal of deprecated APIs and changes to default configuration values. The Spark 4.0 migration assistant, a new tool included in the release, analyzes existing Spark code and identifies compatibility issues, significantly reducing the effort required to migrate large codebases.
