A high-performance I/O architecture for distributed data processing

Crail is a user-level I/O architecture for the Apache data processing ecosystem, designed from the ground up for high-performance networking and storage hardware. With Crail, hardware performance advantages become visible at the application level and translate into workload runtime improvements.

Effectively leveraging fast networking and storage hardware for distributed data processing remains challenging. Often the hardware integration takes place too low in the stack and, as a result, performance advantages are overshadowed by higher-layer software overheads. Moreover, new opportunities for fundamental architectural changes within the data processing layer are not being explored.

Crail enables data processing frameworks like Spark or Tez to operate efficiently on modern hardware. It also enables new configurations that previously were difficult to operate efficiently such as storage and memory disaggregation, horizontal storage tiering and heterogeneous clusters.

In recent years, we have seen major performance improvements in both networking and storage hardware. Network interconnects and fabrics have evolved from 1–10 Gbps bandwidth ten years ago to 100 Gbps today.

More about   Crail logo

Storage hardware has undergone similar improvements while moving from spinning disks.These hardware changes have also prompted the need for new I/O interfaces. Traditional operating system abstractions such as sockets for networking or the block device for storage proved to be insufficient to make the rich semantics and high performance of the hardware available to applications. Instead, new interfaces such as RDMA1 or NVMe have emerged. These interfaces enable user-level hardware access and asynchronous I/O and allow applications to take full advantage of the high-performance hardware.