A high-performance I/O architecture for distributed data processing
Apache Crail is a user-level I/O architecture for the Apache data processing ecosystem, designed from the ground up for high-performance networking and storage hardware. With Apache Crail, hardware performance advantages become visible at the application level and translate into workload runtime improvements.
Effectively leveraging fast networking and storage hardware for distributed data processing remains challenging. Often the hardware integration takes place too low in the stack and, as a result, performance advantages are overshadowed by higher-layer software overheads. Moreover, new opportunities for fundamental architectural changes within the data processing layer are not being explored.