Apache DataFusion

This is interesting: the smallest #Sail instance on ClickBench using c6a.2xlarge (8 vCPUs and 16 GiB of memory) outperforms every #Spark and Spark accelerator configuration across all instance sizes. The best-performing Spark setup is the Comet Accelerator on c7a.metal-48xl (192 vCPUs and 384 GiB of memory). It’s wild that Sail, running on just 8 vCPUs and 16 GiB, still significantly outperforms an accelerated Spark workload running on 192 vCPUs and 384 GiB. Notably, both Sail and Comet are powered by #DataFusion. Benchmark: https://lnkd.in/gSMjxrtR Sail: https://lnkd.in/gQiakuU6

I'm pleased to announce the release of Apache DataFusion Comet 0.12.0, a high-performance accelerator for Apache Spark built on the DataFusion query engine. This release represents four weeks of development with 105 merged PRs from 13 contributors, delivering significant improvements across performance, functionality, and developer experience. Key highlights include experimental native Apache Iceberg scan support through iceberg-rust integration, enabling unmodified Iceberg Java to handle query planning while Comet provides native execution acceleration. We've also introduced new SQL functions including concat, abs, sha1, and hyperbolic trigonometric functions, along with the CometLocalTableScanExec operator for native local table scan support. The release features substantial code architecture improvements with unified operator serialization and refactored expression handling, making it easier for contributors to extend functionality. Configuration has been simplified with better environment variable support and streamlined memory management settings. We've addressed numerous bug fixes for string operations, join handling, and null value processing, while updating to Spark 3.5.7 and DataFusion 50.3.0. Documentation has been enhanced with improved contributor guides and consistent formatting. Comet continues to support Spark 3.4.3, 3.5.x, and 4.0.1 across multiple JDK and Scala versions. Read the blog post for more details. https://lnkd.in/gn3jWyen

The Apache #datafusion is excited to congratulate the Kubeflow Trainer project on the launch of its new Distributed Data Cache to accelerate data loading and maximize GPU utilization in ML training and optimization jobs on Kubernetes, now powered by #datafusion. 🎉 🚀 It's inspiring to see our query engine helping accelerate scalable, Cloud Native AI workloads in such a forward-looking project. We're grateful to the Kubeflow community for the collaboration and can't wait to see the innovations this unlocks for AI practitioners everywhere. Here's to faster pipelines, smoother iteration, and a growing ecosystem built on open data! Please find the Kubeflow Trainer project https://lnkd.in/gjQgUJhQ https://lnkd.in/gFvhm56K Thanks Akshay Chitneni Rus P. and Andrey Velichkevich for the collaboration from #kubeflowtrainer #datafusion #kubeflow #kubeflowtrainer #cloud #aiml #distributed

Accelerating Apache Spark's Execution Engine There’s been some serious work in the Spark performance space - particularly around optimizing its "execution engine" Let’s take a step back. A typical query engine consists of 5 key components: - Language frontend (e.g., SQL parser) - Intermediate Representation (IR) - Optimizer - Execution Engine (where the real computation happens) - Execution Runtime (task coordination, fault-tolerance) For years, Spark has relied on JVM-based execution - flexible and scalable, but not always the most efficient for CPU-bound tasks. To mitigate some of this overhead, Spark 2.0 introduced Whole Stage Code Generation, replacing the Volcano iterator model with Java bytecode generation This key improvement brought up to 2x speedups for many query workloads. But recent work is pushing this even further! By introducing native vectorized execution, using projects like Apache Arrow for fast in-memory columnar processing. Here are 3 cutting-edge efforts focused on accelerating Spark’s execution layer with high-performance native engines: ✅ Velox: - A C++ native execution engine optimized for performance & modularity. - Instead of every system reinventing the execution layer, Velox offers reusable components for scan, filter, join, aggregation, etc. - Advanced optimizations like SIMD, lazy evaluation, and adaptive query execution ✅ Apache Gluten: - Gluten acts as the JNI bridge between Spark SQL and native engines like Velox. - Spark’s logical plan stays intact but execution is offloaded to Velox. - Removes bottlenecks of JVM execution ✅ Apache DataFusion Comet: - An integration of Apache DataFusion (Rust-based query engine) into Spark - Spark’s execution engine is replaced with DataFusion’s native path - No code changes needed - zero-friction drop-in acceleration The shift from JVM-bound execution to native vectorized engines is already showing measurable impact in production. We're entering a new phase where modular, embeddable native engines are going to accelerate the next generation of open lakehouse compute. Some helpful videos in comments! #dataengineering #softwareengineering

Nga Tran Carl Yeksigian and I are organizing the next Apache DataFusion meetup next Wednesday Nov 12 in Boston. Signup here: http://lu.ma/w9pw5rce

Save the date -- Wednesday July 22, 2026 for the first Apache DataFusion meetup in Denver: https://luma.com/jsu6faie

I have been following the amazing work that has been going on in the Apache DataFusion Comet project 🚀 Yesterday, the 0.11.0 version was released. Highlighting some of these new/improved stuff. Key is to watch the 'type' of PR landing in the project to tackle a diverse set of things in the data systems space. ✅ Native Parquet Modular encryption support: - meaning even when Apache Spark offloads query execution to Comet’s native (Rust) runtime, it can still decrypt encrypted Parquet files securely - uses Apache Arrow-rs & Datafusion under the hood ✅ Improved Spark support: - version 4.0.1 - enhances ANSI mode support, adding full compliance for arithmetic, integral divide, rounding & remainder operations ✅ Apache Iceberg: - 1.9.1 version support - Adds a new API that removes dependency on Parquet’s ColumnDescriptor, resolving method-not-found errors and enabling smoother Iceberg-Comet integration - Fixes connection leaks observed in production Iceberg workloads ✅ Better Memory Management: - brings huge improvements to memory management, making it easier to deploy and more resilient to out-of-memory conditions Release notes in comment. Highly recommend following the project. #dataengineering #softwareengineering

I'm super proud of the 0.11.0 release for Comet! I think the memory management improvements will help users accelerate their Spark jobs with less time spent tuning, and features like Parquet Modular Encryption are great to see come together through collaboration with Arrow-rs and DataFusion.

This is an excellent example of the value of Apache Arrow & Apache DataFusion as the foundation for building new high-performance specialized databases.