Master’s Thesis or Internship Project
Computational Storage on SmartNIC Accelerators for Cloud Applications
In recent years SmartNICs have become the state-of-the-art solution for providing storage and network virtualization in cloud environments. Leading cloud providers are deploying SmartNIC-based services, like AWS Nitro or Azure SmartNIC, to provide isolation, security, increased performance, and lower cost. While in the past SmartNICs were custom designs by Cloud vendors, they are becoming more widely available. Several hardware vendors offer commercial off-the-shelf (COTS) SmartNICs, like NVIDIA BlueField, Pensando DSC, Xilinx SN, etc. The aim of this project is to exploit SmartNIC hardware to offload storage computational for use cases relevant to a cloud datacenter.
There are a multitude of distributed key-value/object store options in cloud environments available today, e.g. IBM COS, Amazon DynamoDB, Reddis, etcd, etc. While NVMe is the defacto standard to access block storage, unfortunately there is no such generic API for kv stores. Each store employs its own API and application must carefully choose between the alternatives or provide adapters to support different stores. The recently released NVMe 2.0 specification adds command set support for key-value commands and provides a generic API to access key-value NVMe drives. Currently available SmartNICs allow emulating NVMe device to present remote storage as local storage in a performant and secure manner. We believe this could be leveraged to present an emulated device with NVMe KV command set to access a distributed kv store. The goal of the project is to use a state-of-the-art distributed kv store and implement the needed functionality on a SmartNIC to emulate a KV NVMe device. Furthermore, the student should explore how this new API can be leveraged in applications, used in virtualized environment and what unique properties can be provided by the SmartNIC, e.g. client side encryption, compression, etc.
Another approach would be to utilize smartNIC hardware to offload computational functions to accelerate database and analytics services. Operations such as data filtering, aggregation, reorganization, etc., could be offloaded, for example, to a remote distributed storage system through smartNICs in a transparent and efficient manner. Existing software-defined distributed systems like Ceph offer support for custom code to be executed on a particular data object, allowing for such interoperability to be facilitated. Open-source projects like Skyhook, already take advantage of such functionality to offload PostgreSQL database operators into a Ceph cluster. One could expand on prior work to provide similar computational functionality by utilizing smartNIC hardware.
We are inviting applications from students to conduct their master’s thesis work or an internship project at the IBM Research lab in Zurich on this exciting new topic. The research focus will be on exploring storage compute offload for database acceleration and/or distributed kv stores. The ideal candidate should be well versed in distributed systems, operating systems, and have strong programming skills (C/C++). Hands-on experience with distributed key-value stores and/or databases are highly desirable.
IBM is committed to diversity at the workplace. With us you will find an open, multicultural environment. Excellent flexible working arrangements enable all genders to strike the desired balance between their professional development and their personal lives.
How to apply
If you are interested in this exciting position, please submit your most recent curriculum vitae.
For more information on technical questions please contact Jonas Pfefferle (JPF@zurich.ibm.com), Software Engineer.