TierBPF: Page Migration Admission Control for Tiered Memory via eBPF

TierBPF is an eBPF-based mechanism that enhances existing software memory tiering systems by incorporating hardware topology and page size into admission decisions, achieving significant throughput improvements across diverse workloads.

The Gist

Imagine your computer's memory (RAM) is like a giant, high-speed library.

In the past, this library had only one type of shelf: fast, expensive, but very small. As computers got bigger and needed to store more data, engineers added a second type of shelf: slower, cheaper, but huge. This is called Tiered Memory.

The computer's operating system acts as the Librarian. Its job is to decide which books (data pages) go on the fast shelves and which stay on the slow ones. The rule has always been simple: "If a book is read often, put it on the fast shelf. If it's rarely read, leave it on the slow shelf."

TierBPF is a new, super-smart assistant for this Librarian. It fixes two major mistakes the old Librarian was making.

The Two Big Problems

Problem 1: The "Whole Box" Mistake

Imagine a book is actually a giant, 2MB cardboard box containing 512 tiny pamphlets.

• The Old Way: If you only need to read one pamphlet inside the box, the old Librarian would move the entire 2MB box to the fast shelf.
  • The Result: You wasted a huge amount of fast shelf space and bandwidth moving 511 pamphlets you didn't need, just to get to the one you did.
• The TierBPF Way: TierBPF says, "Wait! Let's open the box." It realizes you only need a small chunk. It cuts the box open and moves only the specific pamphlets you need.
  • The Result: It saves space and speed. It's like ordering a single slice of pizza instead of the whole 2-foot pie just because you're hungry.

Problem 2: The "One-Size-Fits-All" Mistake

Imagine the library has two different delivery trucks:

1. Truck A (CXL): Has two lanes. One lane is for picking up books (Read), and the other is for dropping them off (Write). They don't interfere with each other.
2. Truck B (PMEM): Has only one lane. It has to do both picking up and dropping off on the same road.

• The Old Way: The Librarian used the exact same strategy for both trucks. "Move the hot books!"
• The Problem: On Truck A (two lanes), if you only move "drop-off" books, you clog the drop-off lane while the pick-up lane sits empty. You could have moved "pick-up" books instead to keep things balanced. On Truck B (one lane), trying to balance lanes is useless because there's only one road; you just need to move the most important books.
• The TierBPF Way: TierBPF looks at the specific truck and the current traffic.
  • If it's Truck A and the "drop-off" lane is jammed, it says, "Let's hold back the drop-off books and move the pick-up books instead."
  • If it's Truck B, it ignores the lanes and just moves the most urgent books.

How Does TierBPF Do This? (The Magic Trick)

You might ask, "How does the assistant know which pamphlets are hot without checking every single one (which would be too slow)?"

TierBPF uses a clever trick called eBPF. Think of eBPF as a security camera inside the library that runs directly on the camera's brain, not on a computer in the back office.

• Old Way: The camera sends a video feed to the back office, a human watches it, writes a report, and tells the Librarian what to do. This is slow and clutters the office.
• TierBPF Way: The camera analyzes the video right there and instantly flashes a green light to the Librarian: "That pamphlet is hot!" It happens so fast it doesn't slow down the library at all.

The Results

The researchers tested this new assistant in a real data center (a giant library).

• Speed Boost: In some cases, the library ran 75% faster. On average, it was about 17% faster.
• Less Waste: It stopped wasting space on the fast shelves with cold, unused data.
• Smarter Decisions: It figured out exactly how big a "box" to move based on how busy the library was. If the library was quiet, it moved big boxes. If the library was chaotic and crowded, it moved tiny, precise chunks.

The Bottom Line

TierBPF is like upgrading a clumsy, rule-following robot librarian into a smart, adaptable human. It doesn't just follow the rule "move hot stuff"; it looks at how the stuff is packed, what kind of truck is delivering it, and how busy the road is, to make the perfect decision every time.

Technical Summary

1. Problem Statement

Modern data centers are increasingly adopting tiered memory architectures (pairing fast, limited DRAM with slower, high-capacity tiers like CXL or PMEM) to address the growing gap between core counts and local memory capacity. Existing software-based tiering systems rely on a high-level abstraction: migrating "hot" (frequently accessed) pages to fast memory and keeping "cold" pages in slow memory.

The authors identify two critical flaws in this abstraction that degrade performance:

1. Page Size Granularity Mismatch: Modern systems often use Transparent Huge Pages (THP) (2 MB) to reduce Translation Lookaside Buffer (TLB) misses. However, existing tiering systems treat a 2 MB THP as an atomic unit. If only a small fraction (e.g., 64 KB) of a 2 MB THP is hot, the system must either:
   • Migrate the entire 2 MB block, wasting fast-tier capacity and bandwidth with cold data.
   • Split the THP entirely into 4 KB base pages, which preserves bandwidth but destroys TLB efficiency, causing performance degradation.
   • Gap: No existing system dynamically selects intermediate page sizes (e.g., 64 KB, 128 KB) during migration.
2. Architecture-Agnostic Policies: Tiering systems apply uniform policies regardless of the underlying hardware.
   • CXL Memory: Uses full-duplex links (independent read/write channels).
   • Intel Optane PMEM: Uses half-duplex DDR links (shared read/write bus).
   • Gap: A policy that rebalances read/write traffic to optimize full-duplex CXL links may be counterproductive or useless on half-duplex PMEM, yet current systems do not adapt to these architectural differences.

2. Methodology: TierBPF

TierBPF is a software mechanism that sits between the memory tiering decision logic and the actual page migration mechanism. It uses eBPF (Extended Berkeley Packet Filter) to implement custom, low-overhead admission control policies.

Core Architecture

TierBPF inserts three eBPF hooks into the kernel's NUMA migration path:

1. Split-Size Selection: Determines the optimal intermediate page size (mTHP) for migration.
2. Subpage Classification: Identifies which specific subpages within a THP are hot or cold.
3. Migration Admission: Decides whether to proceed with migration based on real-time hardware traffic patterns (e.g., read/write dominance).

Key Technical Components

A. Lightweight Memory Profiling
To make decisions without the high overhead of per-page tracking (which scales poorly with working set size), TierBPF uses:

• Global Subpage Histogram: A fixed-size (4 KB) array per application that maps hardware-sampled memory accesses to subpage offsets within a 2 MB THP. It ignores cold THPs (which contribute few samples), focusing only on hot regions.
• In-Kernel eBPF Processing: Unlike previous systems (e.g., MEMTIS) that process samples in user-space (causing context switches and data copying), TierBPF processes hardware performance samples directly in the kernel interrupt context using eBPF.
• Dual Blocked Counting Bloom Filter (bCBF): Used specifically for CXL systems to track read/write ratios per page without maintaining full per-page counters.

B. Contention-Aware mTHP Splitting
TierBPF dynamically selects the page size based on memory contention:

• Low Contention: Prefers larger pages (2 MB) to maximize TLB coverage.
• High Contention: Activates splitting to intermediate sizes (e.g., 64 KB, 128 KB).
• Algorithm: It calculates the "heat density" of subpages. If a specific intermediate size (e.g., 64 KB) captures a high density of hot subpages (e.g., >75%), it selects that size. This balances TLB efficiency with bandwidth conservation.

C. Architecture-Aware Admission Control
TierBPF adapts migration decisions to the hardware duplex mode:

• CXL (Full-Duplex): If read traffic is the bottleneck, the system prioritizes migrating read-heavy pages and defers write-heavy pages to keep channels balanced.
• PMEM (Half-Duplex): Read/write rebalancing is disabled; all hot pages are admitted regardless of type, as they share the same bus.

3. Key Contributions

1. Identification of Limitations: The paper formally identifies the lack of migration-time page-granularity awareness and architecture-aware filtering in current tiering systems.
2. TierBPF System: The first system to support dynamic, contention-aware mTHP size selection and memory duplex mode-aware admission control.
3. eBPF Implementation: All policies are implemented as user-customizable eBPF programs, decoupling policy from mechanism and allowing deployment without kernel source modifications.
4. Scalable Profiling: Introduces a lightweight profiling mechanism (global histogram + in-kernel eBPF) that is independent of the application's working set size, solving the scalability issues of previous approaches like MEMTIS.

4. Evaluation Results

The authors evaluated TierBPF on two hardware platforms (CXL memory expansion and Intel Optane PMEM) integrated with three tiering systems: AutoNUMA, TPP, and Colloid.

• Performance Gains:
  • Across 17 memory-intensive workloads, TierBPF achieved a geometric mean throughput improvement of 7.4% to 17.7% over baseline systems.
  • Individual workloads saw improvements of up to 75%.
  • On the TPP system, TierBPF achieved a 17.7% geometric mean gain.
• Comparison with MEMTIS:
  • TierBPF outperformed the state-of-the-art MEMTIS system by up to 26%.
  • MEMTIS, which splits THPs only into 4 KB pages, suffered from high TLB miss rates (up to 47% higher than TierBPF in some cases), whereas TierBPF's intermediate mTHP sizes preserved TLB efficiency.
• Migration Success Rate:
  • By splitting 2 MB pages into smaller chunks (e.g., 64 KB), TierBPF drastically reduced migration failures caused by memory fragmentation. For some workloads, migration failures dropped by 777x (e.g., from 8.5M to 11K failures).
• Admission Control Impact:
  • Under high memory contention (20 GB/s), the traffic-balancing admission control provided modest gains or was neutral, as total bandwidth was the bottleneck.
  • Under moderate contention (10 GB/s), the admission control provided additional gains (up to 5.2%) by optimizing channel utilization on CXL.

5. Significance

TierBPF represents a significant shift in how operating systems manage tiered memory. By leveraging eBPF, it transforms memory tiering from a static, abstraction-heavy process into a dynamic, hardware-aware, and programmable system.

• Practicality: It solves the "all-or-nothing" dilemma of THP migration, allowing systems to migrate exactly the amount of data needed without sacrificing TLB performance.
• Adaptability: It enables a single software stack to optimize performance across diverse and emerging hardware (CXL, PMEM, future disaggregated memory) without code changes.
• Efficiency: It demonstrates that low-overhead, in-kernel profiling can replace heavy user-space monitoring, making advanced memory management feasible in production environments.