Towards Safe, Flexible, and Easy Software Compartmentalisation

Abstract

Exacerbated by the ubiquity of software, software vulnerabilities increasingly threaten individuals, businesses, and critical infrastructure. One of the root causes of software insecurity is monolithic software design. By designing programs as single units of trust and privilege, the compromise of a single dependency, module, or software part instantly leads to that of the entire program. Software compartmentalisation provides one way to address this problem. As a software engineering practice where developers break down programs into groups of isolated and distrusting compartments, software compartmentalisation ensures that each part of a program only runs with the smallest set of necessary privileges. In the event of a compromise, compartmentalisation contains the exploit, forcing attackers to deploy significantly more complex attacks to breach software systems. Yet, despite having proved its worth in the field, software compartmentalisation is still not a popular practice: outside major software applications such as web browsers, mainstream programs remain vastly monolithic. We are missing out on the benefits of software compartmentalisation, at a time when we need them more than ever. This thesis makes three steps towards mainstream software compartmentalisation. In a first part, we investigate how to make compartmentalisation faster and safer by specialising compartmentalisation and protection profiles of entire systems towards application workloads. To achieve this we propose FlexOS, a modular operating system which enables users to easily implement highly-specialised compartmentalisation policies, along with a semi-automated method to explore this design space. We show that FlexOS opens for a vast design space, and that this modularity does not come at the cost of performance compared to existing OSes with fixed safety configurations. In a second part, we investigate interface vulnerabilities, the confused deputy vulnerabilities which arise at insufficiently hardened compartment boundaries. We taxonomise these issues, showing that there exist no complete mitigations, and that they affect all known compartmentalisation approaches. We propose ConfFuzz, a fuzzer specialised to detect interface vulnerabilities at possible compartment boundaries, and use it to gather a wide data-set of 629 potential vulnerabilities in real-world software. Systematically studying these issues, we show, among others, that not all interfaces are affected similarly, that API size is uncorrelated with the prevalence of interface vulnerabilities, and that addressing interface vulnerabilities goes beyond writing simple checks. In a third part, we perform a a large-scale systematisation of knowledge in the field of software compartmentalisation. By identifying and framing existing trends and approaches in software compartmentalisation, along with instances of compartmentalisation that made it into the mainstream, we aim to provide insights on the challenges we still need to tackle to bring the benefits of software compartmentalisation to the mainstream. We show that popularising software compartmentalisation and bringing research advances to the mainstream will require progress towards eliminating the need for developers to manually define compartmentalisation policies (or, when relevant, helping them doing so); towards better framing separation costs early on; to designing abstractions that will stand the test of time and progress; and to better challenging our threat models, particularly in light of interface safety issues.

Date of Award	8 May 2024
Original language	English
Awarding Institution	The University of Manchester
Supervisor	Pierre Olivier (Main Supervisor) & Giles Reger (Co Supervisor)