Apache Crail
A high-performance I/O architecture for distributed data processing
Apache Crail is a user-level I/O architecture for the Apache data processing ecosystem, designed from the ground up for high-performance networking and storage hardware. With Apache Crail, hardware performance advantages become visible at the application level and translate into workload runtime improvements.
Effectively leveraging fast networking and storage hardware for distributed data processing remains challenging. Often the hardware integration takes place too low in the stack and, as a result, performance advantages are overshadowed by higher-layer software overheads. Moreover, new opportunities for fundamental architectural changes within the data processing layer are not being explored.
Apache Crail enables data processing frameworks like Apache Spark or Apache Tez to operate efficiently on modern hardware. It also enables new configurations that previously were difficult to operate efficiently such as storage and memory disaggregation, horizontal storage tiering and heterogeneous clusters.
In recent years, we have seen major performance improvements in both networking and storage hardware. Network interconnects and fabrics have evolved from 1–10 Gbps bandwidth ten years ago to 100 Gbps today.
Storage hardware has undergone similar improvements while moving from spinning disks.These hardware changes have also prompted the need for new I/O interfaces. Traditional operating system abstractions such as sockets for networking or the block device for storage proved to be insufficient to make the rich semantics and high performance of the hardware available to applications. Instead, new interfaces such as RDMA or NVMe have emerged. These interfaces enable user-level hardware access and asynchronous I/O and allow applications to take full advantage of the high-performance hardware.
Crail, Tez, and Spark are trademarks of the Apache Software Foundation.
