This Week in Ballista #407 Feb 2021
Welcome to the fourth edition of “This Week in Ballista”, a weekly newsletter that summarizes activity in the Ballista Distributed Compute project.
Ballista is a modern distributed compute platform powered by Apache Arrow and primarily implemented in Rust, but designed to provide first-class support for other programming languages, including Python, C++, and Java.
Distributed Query Execution
Distributed query execution is now implemented and has been tested successfully end to end but there is still a little more work to do before this is enabled by default and able to support the benchmarks.
The source for the query planner is here. This is a simple implementation that will be optimized once the benchmark queries are functionally working.
The query planner works by translating a DataFusion physical query plan into a Ballista physical query plan by breaking the plan up into query stages whenever partitioning changes between a parent and child operator. A query stage is represented by the QueryStageExec operator.
A query stage can be executed by distributing the execution across the available executors in the cluster. The output from each partition is streamed to disk on the executor in Arrow IPC format. Future query stages then retrieve these results using a ShuffleReaderExec operator.
Finally, a CollectExec operator can be executed to retrieve the final result partitions in the client and coalesce them into a single partition.
All of these pieces work and now the following work needs to happen to make this fully usable:
- The scheduler already has the ability to receive a logical plan and execute a distributed query using the available
executors in the cluster, and the Ballista
DataFrame::collectmethod now needs to be updated to use this mechanism instead of sending the logical plan to one executor to be executed in-process with DataFusion (#485).
- There is still a little more work to do in the serde module to add support for all of the operators and expressions required to support the benchmark queries (multiple issues).
- The User Guide needs updating to reflect the changes in 0.4.0 (#486).
At the current rate of progress, it is very likely that there will be a 0.4.0 alpha release before the end of the month and at this point it will be easier to have more contributors working on different areas of the project.
There is now a @BallistaCompute Twitter account. Follow this account to receive notifications when “This Week in Ballista” is posted each week. Eventually there will be an RSS feed and support for email notifications as well.
There has been an effort to start cleaning up the GitHub issues and more issues are now labeled with help wanted.
There are a mix of small short term issues and larger longer term initiatives.
Join the Community
There is a growing community in the Ballista Discord Channel. This is a great place to ask questions and learn more about the project.