How AWS’s Firecracker virtual machines work

Since 2014, Amazon Web Services (AWS) has been offering “serverless” computing through AWS Lambda. With Lambda, customers don’t have to worry about managing servers or adjusting capacity in response to fluctuating demand. AWS does the provisioning automatically, and customers simply pay for the resources they use.

When we first built Lambda, we had to choose between two security approaches. One, containerization, is fast and resource efficient but doesn’t provide strong isolation between customers; the other, running code inside a virtual machine, offers greater security at the cost of computational overhead. Security is always our top priority at AWS, so we built Lambda using traditional VMs.

Our customers challenged us to offer faster scaling, lower latency, and advanced features like provisioned concurrency. We knew we couldn’t build those features on traditional VMs, so we built Firecracker, which we released in November 2018 as an open-source virtualization platform.

Firecracker offers the best of both worlds: the security of hardware-virtualization-based virtual machines and the resource efficiency and fast startup time of containers. Last week, at the USENIX Symposium on Networked Systems Design and Implementation (NSDI ’20), my colleagues and I presented a paper explaining how Firecracker works.

Part of the virtualization stack is the Virtual Machine Monitor (or VMM), which sets up virtualization, manages memory, and handles I/O (like network connectivity and on-disk storage).

Traditional VMMs can be nearly as complex as full operating systems. QEMU, a virtual machine monitor that is commonly used in conjunction with the Linux kernel virtual machine (KVM), has more than 1.4 million lines of code (and a correspondingly broad and powerful feature set).

One reason Firecracker is so much more efficient than a typical virtual machine is its stripped-down VMM, which has only 50,000 lines of code — a 96% reduction over QEMU. This allows us to create a single microVM for the code that each customer program executes in Firecracker, a simple but strong security model. Moreover, those 50,000 lines of code are written in the Rust language, which is notable for its built-in security and correctness features. A single server can create up to 150 Firecracker microVMs per second and run thousands of microVMs at the same time.

Of course, drastically reducing the size of the VMM reduces its functionality just as drastically. Firecracker doesn’t implement traditional devices like a BIOS or PCI bus and instead communicates with the guest kernel through optimized virtio interfaces. Where a typical virtualization environment simulates the behavior of the machine that a program thinks it’s running on, with virtio, the program knows that it’s running on a simulation. This enables it to cooperate better with the virtual machine, making execution more efficient.

We also know that serverless workloads don’t need hardware features like USB, displays, speakers, and microphones, so we simply didn’t implement support for any of these. Traditional VMMs, which target both desktop and cloud use cases, need to include all this complexity.

Firecracker powers the AWS Lambda service, where it currently handles trillions of requests each month for hundreds of thousands of AWS customers.