Aquifer: Hierarchical Memory Pooling with CXL and RDMA for MicroVM Snapshots

Aquifer is a novel hierarchical memory pooling system that combines low-latency CXL and cluster-wide RDMA to optimize MicroVM snapshot restoration by employing a hotness-based format, an ownership-based coherence protocol, and a hybrid copy-and-demand paging mechanism, thereby significantly reducing cold-start latency compared to existing solutions.

The Gist

Imagine a massive cloud data center as a giant hotel with thousands of rooms (servers). Each room has a specific amount of "memory" (like a desk space for the computer to work on).

The Problem: The "Empty Desk" Waste
In this hotel, a strange thing happens. Often, a room's CPU (the brain) is fully busy, but the desk space (memory) assigned to it is sitting empty because the guest hasn't used it yet. The hotel owners call this "memory stranding." They are paying for thousands of desks that no one is sitting at. In fact, 25–35% of all the desks in the hotel are wasted.

The Solution: A Shared Workspace
To fix this, the hotel decides to build a giant, shared workspace in the basement where everyone can pull desks from when they need them. This is called "memory pooling."

However, there are two types of shared workspaces, and neither is perfect on its own:

1. The "Super-Fast" Local Shelf (CXL): This is a shelf right next to your room. You can grab a book from it instantly, like reaching for something on your own nightstand. But, it's small. It can only serve a few rooms in your immediate hallway (a "pod").
2. The "Huge" Basement Archive (RDMA): This is a massive warehouse in the basement that holds millions of books. It can serve the entire hotel. But, it takes longer to walk down there, get the book, and bring it back. Plus, you have to fill out paperwork (software overhead) every time you go.

The Innovation: Aquifer
The researchers built a system called Aquifer that combines both. It uses the "Super-Fast Shelf" for the books you need right now and the "Huge Basement" for the books you might need later or never need at all.

Here is how Aquifer works, using simple analogies:

1. The "Smart Snapshot" (Cleaning the Book)

When a guest (a MicroVM) leaves a room, the hotel takes a "snapshot" (a photo) of their desk to save it for later.

• The Old Way: They took a photo of the entire desk, including empty spaces, dust, and blank pages. This wasted huge amounts of storage.
• Aquifer's Way: They analyzed these snapshots and found that 83% of the desk was just empty space (zeros) and another 60% of the non-empty space was never touched by the guest.
• The Fix: Aquifer throws away the empty space entirely. It puts the "hot" books (the ones the guest actually used) on the Super-Fast Shelf and the "cold" books (the ones they didn't touch) in the Huge Basement. This saves massive amounts of space.

2. The "Traffic Cop" (Keeping Order)

Since multiple guests might want to read the same "hot" books from the Super-Fast Shelf at the same time, there's a risk of chaos. If one guest changes a page, another might see the old version.

• The Problem: The Super-Fast Shelf (CXL 2.0) doesn't have a built-in "traffic cop" to tell everyone when a book has been updated.
• The Fix: Aquifer invented a new rulebook. It treats the "Pool Master" (the librarian) as the only person allowed to write or update the books. Everyone else is just a "borrower" who can only read.
• The Mechanism: Before a guest grabs a book, they check a counter. If the librarian is trying to update the book, the guest waits. If the book is safe, the guest grabs it, reads it, and then updates the counter to say, "I'm done." This ensures everyone sees the correct version without needing expensive hardware help.

3. The "Pre-Loading" Trick (Speeding Up the Start)

When a guest returns to their room, the hotel needs to set up their desk quickly so they can start working.

• The Old Way: The guest would sit down, reach for a book, realize it's missing, run to the basement, get it, and come back. This happens for every single book, causing a long delay (the "cold start").
• Aquifer's Way:
  • Step 1: Before the guest even sits down, the hotel staff runs to the Super-Fast Shelf and copies all the "hot" books directly onto the guest's desk. This happens instantly.
  • Step 2: If the guest reaches for a "cold" book (one that was in the basement), the staff sends a request to the basement in the background while the guest starts working on the books that are already there. The guest never has to wait for the basement delivery to start working.

The Results

The researchers tested this system on a simulated version of their hotel.

• Speed: Aquifer was 2.2 times faster at getting guests started compared to the standard method (Firecracker).
• Efficiency: It was also 1.3 times faster than the next best attempt at solving this problem.

In Summary
Aquifer is like a smart hotel manager who realizes that most guests only use a few specific items on their desk. By throwing away the empty space, keeping the most-used items on a super-fast shelf, and pre-loading those items before the guest arrives, they eliminated the long wait times and wasted space that plagued the cloud data center.

Technical Summary

Technical Summary: Aquifer

Problem Statement

Modern cloud data centers suffer from memory stranding, where provisioned DRAM sits idle because co-located CPU cores are fully subscribed. Measurements indicate this wastes 25–35% of installed memory capacity in production clusters. While memory pooling via RDMA and CXL offers a remedy, neither technology alone suffices for MicroVM-based serverless computing:

• RDMA provides cluster-wide reach but incurs significant software overhead (page faults, kernel intervention) on every remote access, making it unsuitable for the latency-critical path of MicroVM cold starts.
• CXL offers low-latency, load/store-transparent access but is currently limited to a single pod (8–16 sockets) and, in CXL 2.0 implementations, lacks hardware-managed cache coherence across hosts.

Consequently, there is no existing solution that combines these tiers to serve MicroVM snapshots efficiently. Current snapshot-based warm restore systems focus on single-machine optimizations or flat pools, failing to address the heterogeneity of a hierarchical CXL+RDMA architecture or the specific semantics of MicroVM sandboxes (which cannot use fork-based cloning like cgroup containers).

Methodology

The authors present Aquifer, the first system to serve MicroVM snapshots from a hierarchical memory pool combining a pod-local CXL tier and a cluster-wide RDMA tier. The design addresses three core challenges:

1. Hotness-Based Snapshot Format

Through a characterization of 9 representative serverless workloads, the authors found that 82.8% of snapshot pages are zero, and of the remaining non-zero pages, 72.7% are "cold" (never accessed during a typical invocation window). Only a small fraction constitutes the "hot" working set.

• Zero Page Elimination: Zero pages are excluded from storage entirely.
• Tiered Partitioning: Non-zero pages are split based on access patterns:
  • Hot Set: Dirtied and read-only pages are stored in the CXL pool for sub-microsecond access.
  • Cold Set: Non-zero pages never accessed during profiling are stored in the RDMA pool to harvest stranded memory capacity.
• Offline Profiling: The system replays sampled invocation requests to identify the hot working set before generating the snapshot, avoiding overhead on production invocations.

2. Ownership-Based Coherence Protocol

Since CXL 2.0 Multi-Headed Devices (MHDs) lack inter-host cache coherence, Aquifer implements a software protocol to ensure correctness when multiple hosts concurrently restore from the same snapshot:

• Immutability: Snapshot data in the CXL pool is read-only for orchestrators (borrowers). The Pool Master is the sole owner responsible for updates/deletions.
• Atomic Borrowing: Borrowers atomically increment a reference count and verify the snapshot state (published vs. tombstone) using Compare-and-Swap (CAS) operations.
• Cache Invalidation: Upon successful borrowing, orchestrators explicitly flush cache lines (clflushopt) to ensure they fetch the latest data from CXL memory, preventing stale reads.

3. Copy-Based Page Serving

The compact, non-file-backed snapshot format precludes standard mmap-based page installation. Aquifer uses a copy-based mechanism via userfaultfd:

• Hot Set Pre-Installation: Before resuming the MicroVM, the orchestrator pre-installs all hot pages from the CXL pool directly into the guest address space using uffd.copy(). This eliminates page faults for the majority of accesses during execution.
• Asynchronous Cold Paging: Cold pages are demand-paged only when the guest faults. An asynchronous handler fetches these pages from the RDMA pool into a local buffer and installs them via uffd.copy(), decoupling fault handling from RDMA completion to avoid head-of-line blocking.

Key Contributions

1. Characterization: A detailed analysis of MicroVM snapshot images revealing that the hot working set is small (5.5% of total pages) and highly fragmented, motivating a compact, hotness-aware format.
2. System Design (Aquifer): A hierarchical CXL+RDMA pooling system integrated with the Firecracker MicroVM monitor, featuring:
   • A hotness-based snapshot format that eliminates zero pages and partitions data across tiers.
   • An ownership-based coherence protocol enabling concurrent access on CXL 2.0 hardware without hardware coherence.
   • A copy-based page serving mechanism that pre-installs hot pages and asynchronously demand-pages cold pages.
3. Evaluation: Implementation and testing on CloudLab hardware (emulating CXL via remote NUMA and RDMA via SR-IOV).

Results

Evaluated across nine serverless workloads (e.g., chameleon, image, recognition), Aquifer demonstrates:

• Performance: A 2.2× geometric-mean speedup in end-to-end invocation time compared to the baseline Firecracker (demand-paging from RDMA).
• Comparison: A 1.3× speedup over FaaSnap (a layered memory mapping approach) and 1.1× over REAP (record-and-prefetch).
• Scalability: The system scales effectively to 32 concurrent MicroVM restores, maintaining performance advantages in most workloads.
• Overhead: The setup phase is faster than alternatives due to the elimination of zero pages and efficient CXL pre-installation, while execution time is drastically reduced by avoiding RDMA latency for the hot working set.

Significance

The paper claims that Aquifer demonstrates that a hierarchical CXL+RDMA pooling architecture is a practical substrate for latency-sensitive cloud workloads. By leveraging the low-latency access of CXL for the critical working set and the high capacity of RDMA for cold data, the system resolves the trade-off between performance and scalability inherent in single-tier pooling. Furthermore, it proves that MicroVM snapshots can be served efficiently from shared memory pools even in the absence of hardware cache coherence, provided a robust software coherence protocol is employed. This approach directly addresses the memory stranding problem in serverless computing while mitigating the cold-start bottleneck.