Overview

Operating systems play a central role in securing most of today’s systems. In particular, cloud services often rely on the OS to enforce isolation between tenants. For instance, containers are increasingly used in PaaS clouds, and tenant isolation is therefore hinged on the trustworthiness of the OS kernel.

We explore novel mechanisms that improve the security of the OS kernel in a practical and efficient way. Examples of such techniques include kernel attack surface reduction, kernel hardening, as well as static and dynamic analysis techniques to find vulnerabilities in OS kernels.

Split kernel

Split Kernel makes kernel hardening more practical by allowing the enablement and disablement of run-time kernel modifications at process (or kernel-thread) granularity. This enables the development of kernel self-protection mechanisms that may have been previously disregarded for being unlikely to be efficient and practical.

We have implemented three kernel hardening mechanisms through assembly rewriting: stack exhaustion prevention, stack clearance to prevent uninitialized stack variable-related vulnerabilities, and kernel function pointer protection.

For instance, when Split Kernel is not used, these hardening mechanisms impose a 33% overhead on SSH connection times. This performance overhead would be considered too high for practical deployment. With Split Kernel, we can choose to enable these kernel hardening mechanisms only when the SSH daemon is running sandboxed. In that case, the overhead is reduced to less than 1%, which is negligible and makes the use of these hardening mechanisms practical.

Our CCS 2014 paper contains many more use cases, as well as an explanation of the overall design, technical details of the implementation, and security considerations related to the use of Split Kernel (in most cases, the performance benefits come without any security drawbacks).

OS kernel overview

Quantifying kernel attack surface

Conventional wisdom dictates that general-purpose commodity OS kernels such as Linux offer a very large attack surface, and that it is necessary to reduce it to improve kernel security.

We find that the kernel attack surface, and how one can quantify it, however, is often ill-defined. In our opinion, this hinders the development of attack-surface reduction mechanisms. Indeed, without an objective way of comparing their effectiveness, it is not possible to improve existing mechanisms. We bridge this gap in our NDSS 2013 paper.

For a given set of assumptions on the operation of the kernel and on the attacker’s interaction with the kernel (e.g., the attacker controls an unprivileged process), we formally define what the attack surface represents. In turn, we use this to derive attack surface metrics that are used to measure and compare attack surfaces.

We perform attack surface measurements not only on current distribution kernels, but also for kernels with kernel attack surface reduction mechanisms we have developed.

Attack surface measurement

Kernel tailoring

Compile-time kernel tailoring is a kernel protection technique that aims to reduce attack surface. We show it is possible to generate a set of kernel configuration options automatically for a given workload via kernel tracing, and that the resulting kernel can be shown to provide a smaller attack surface.

Further details can be found in our HotDep’12 and NDSS’13 papers.

OS kernel overview

Kernel trimming

Run-time kernel trimming also aims to reduce the attack surface, but at a finer granularity and without requiring the kernel to be recompiled. Selectively instrumenting kernel functions also incurs minimal performance overhead.

Evaluation results show that the attack surface can be reduced significantly more than via tailoring.

Further details can be found in our Eurosec’11 and DIMVA’14 papers.

OS kernel overview

Ask the experts

Anil Kurmus

Anil Kurmus

IBM Research scientist

Matthias Neugschwandtner

Matthias Neugschwandtner

IBM Research scientist

Alessandro Sorniotti

Alessandro Sorniotti

IBM Research scientist