DAXFS: A Lock-Free Shared Filesystem for CXL Disaggregated Memory

This paper presents DaxFS, a novel Linux filesystem that leverages CXL's hardware cache coherence and atomic operations to enable lock-free, decentralized multi-host coordination and high-performance shared memory access without centralized bottlenecks.

The Gist

The Big Picture: The "Shared Whiteboard" Problem

Imagine a group of friends (servers) who want to work on a giant, shared whiteboard (memory) in a room.

• The Old Way (Traditional Filesystems): Usually, if two people want to edit a document, they have to send it back and forth over the mail (network), or they have to wait in line for a single "manager" to tell them when they can write. This is slow and creates traffic jams.
• The New Hardware (CXL): A new technology called CXL is like a super-fast, magical whiteboard where everyone can see changes instantly, and the hardware itself guarantees that if two people try to write on the same spot at the exact same time, the system sorts it out without them crashing into each other.
• The Problem: Until now, we didn't have a "rulebook" (filesystem) that knew how to use this magical whiteboard efficiently. Existing rulebooks were too slow, required a manager, or made everyone copy the whiteboard onto their own desks (wasting space).

DaxFS is the new rulebook designed specifically for this magical whiteboard. It lets multiple computers write to the same memory at the same time, instantly, without a manager, and without wasting space.

---

Key Concepts & Analogies

1. The "No-Manager" Rule (Lock-Free Coordination)

In a normal office, if two people want to edit the same spreadsheet, one has to lock the file, edit it, and unlock it. The other person has to wait.

• DaxFS Approach: Imagine a group of friends playing a game where they can all write on the whiteboard simultaneously. If two people try to write on the same square, they use a special "magic pen" (called cmpxchg). The pen only works if the square is empty. If it's already taken, the person just tries the next square.
• The Result: No one ever has to wait in line. There is no "manager" telling people when to write. It's pure, chaotic, but perfectly organized speed.

2. The "Smart Filing Cabinet" (The Hash Overlay)

How do you find a specific file on a whiteboard that has millions of scribbles?

• Old Way: You might have to walk down every row of the whiteboard looking for the name.
• DaxFS Approach: DaxFS uses a Hash Overlay. Think of this as a giant, magical index card system.
  • You want to find "File A"? You do a quick math trick to know exactly which drawer it's in.
  • If the drawer is empty, you drop your file in.
  • If the drawer is full, you try the next one immediately.
  • Because everyone uses the same magic math and the same "magic pen," multiple people can drop files into different drawers at the exact same time without bumping into each other.

3. The "Shared Library" (Cooperative Caching)

Usually, if Computer A reads a file, it keeps a copy in its own memory. If Computer B needs the same file, it makes a second copy. This wastes memory.

• DaxFS Approach: Imagine a single, shared library in the middle of the room. Everyone reads from the same book.
• The "MH-Clock" Algorithm: What happens if the library gets too full? Who throws a book out?
  • In normal systems, a librarian (a central daemon) decides what to throw away.
  • In DaxFS, everyone is their own librarian. They use a clever "Clock" system. Each computer looks at the books, checks if anyone is currently reading them (using a "refcount" tag), and if a book hasn't been touched in a while, they gently push it out to make room for a new one. They do this without asking permission, ensuring no one throws out a book someone is currently reading.

4. The "Zero-Copy" Superhighway

Normally, when a program wants data, the computer copies it from the hard drive -> to the system memory -> to the program. It's like moving a box from a truck to a warehouse, then to your house.

• DaxFS Approach: DaxFS builds a direct tunnel. The program points its finger directly at the memory on the whiteboard. No moving boxes. No copying. It's "Zero-Copy."
• Why it matters: This is crucial for AI. If you are loading a massive AI brain (a Large Language Model) into a GPU, DaxFS lets the GPU grab the data directly from the shared memory without the CPU getting in the way.

5. The "GPU" Connection

GPUs (the chips that power AI) are usually very fast but isolated. They usually have to ask the CPU to get data for them.

• DaxFS Innovation: Because DaxFS is built on "magic pens" (atomic operations) that work over the same high-speed wires (PCIe) that connect GPUs, the GPU can actually participate in the filing system!
• The Analogy: Imagine the GPU is a super-fast robot. Instead of waiting for a human (CPU) to fetch a book from the library, the robot can run up, check the index, and grab the book itself. The paper shows this works incredibly fast, limited only by the speed of the wires, not the software.

---

The Results: How Fast Is It?

The researchers tested DaxFS against the current gold standard for in-memory files (called tmpfs).

1. Writing Speed: When multiple computers try to write random data at the same time, DaxFS was 2.68 times faster than the competition. It's like a highway where everyone can merge instantly without traffic lights.
2. Reading Speed: It was also faster at reading, especially when many people are reading at once.
3. Accuracy: Even when two computers tried to write to the exact same spot at the exact same time, DaxFS got it right 99%+ of the time with no lost data.

Why Should You Care?

• For AI: Training AI models requires massive amounts of memory. DaxFS allows multiple computers to share that memory pool without duplicating data, saving money and speeding up training.
• For Containers: If you run 100 containers that all need the same operating system files, DaxFS lets them all share one single copy in memory instead of 100 copies.
• The Future: As CXL hardware becomes common in data centers, DaxFS is the software layer ready to unlock its full potential, turning a "shared memory" dream into a fast, practical reality.

In a nutshell: DaxFS is the traffic cop that doesn't stop traffic. It uses the hardware's superpowers to let thousands of computers write and read from a shared memory pool simultaneously, safely, and incredibly fast.

Technical Summary

1. Problem Statement

The emergence of CXL (Compute Express Link) 3.0 enables multiple independent servers to share byte-addressable memory with hardware-coherent atomic operations (load/store with cache-line granularity). While this creates a new opportunity for near-DRAM latency shared memory, existing storage solutions fail to exploit this capability effectively:

• Per-Host Filesystems (e.g., tmpfs, ext4-dax): Maintain redundant metadata and page caches on each host, negating the memory pooling benefits of CXL.
• Distributed Filesystems (e.g., NFS, CephFS): Introduce network protocol overhead (microsecond latency) that defeats the purpose of nanosecond-scale CXL memory access.
• Existing CXL Filesystems (e.g., FamFS): Support multi-host mounting but enforce a single-master model where clients cannot create, write, or delete files independently; they rely on log replay from a designated host.
• GPU Data Path: Loading large datasets (e.g., LLM weights) into GPUs typically involves redundant copying through the CPU page cache, creating bottlenecks.

Core Challenge: There is no filesystem that enables lock-free, concurrent multi-host writes on CXL shared memory without a centralized coordinator, while maintaining POSIX semantics and zero-copy access.

2. Methodology & Design

The authors propose DaxFS, a Linux filesystem designed specifically for CXL shared memory. Its design is built on three principles: memory is the storage (not a cache), reads are free (direct pointer dereference), and coordination uses only hardware atomics.

Key Architectural Components:

1. Lock-Free Coordination via cmpxchg:

   • DaxFS uses the hardware-supported cmpxchg (Compare-and-Swap) atomic operation as its sole coordination primitive.
   • Since CXL ensures cache coherence across hosts, a cmpxchg issued by Host A is atomic relative to Host B on the same address. This eliminates the need for locks, journals, or centralized coordinators.

2. CAS-Based Hash Overlay:

   • To enable concurrent writes, DaxFS employs an open-addressing hash table stored in DAX memory.
   • Mechanism: Each bucket (16 bytes) contains a state key and a pool offset. Insertions use a cmpxchg loop: if a bucket is FREE, the host attempts to swap it to USED with the new key. If the slot is taken, linear probing resolves collisions.
   • Pool Allocator: Uses an atomic bump pointer for allocation. Metadata and data pages are aligned to ensure contiguous memory for efficient read coalescing.

3. Shared Page Cache with MH-Clock Eviction:

   • A shared page cache resides in DAX memory, accessible by all hosts.
   • MH-Clock Algorithm: A decentralized adaptation of the classic CLOCK algorithm. Instead of a single global hand, each host independently scans a local probe window (8 slots).
   • Eviction Logic: Hosts look for "cold" slots (ref bit=0, refcount=0). If all are hot, they clear the ref bit and yield. A background sweep periodically advances a shared atomic "evict hand" to clear ref bits in windows, ensuring fair decay even under contention.
   • Safety: A 4-bit refcount in the slot metadata ensures slots actively being read are never evicted.

4. Zero-Copy Data Path:

   • Reads resolve directly to pointers in DAX memory via the hash overlay or base image.
   • For mmap, the filesystem installs Page Frame Number (PFN) mappings directly to the physical memory, allowing user-space loads to bypass the kernel page cache entirely.
   • GPU Integration: The filesystem exports the DAX region via dma-buf, allowing GPUs to perform Peer-to-Peer (P2P) reads. GPU threads can also participate in coordination using PCIe AtomicOp TLPs.

5. Flat Directory Format:

   • Directories use a flat array of fixed-size entries (271 bytes) rather than pointer-based trees (like B-trees). This prevents dangling pointers, allows bounded iteration, and enables single-pass validation of untrusted images at mount time.

3. Key Contributions

1. DaxFS Implementation: The first filesystem enabling lock-free multi-host writes on CXL shared memory without a centralized coordinator.
2. MH-Clock Algorithm: A novel, decentralized cache eviction mechanism adapted for lock-free operation across CXL-connected hosts, eliminating the need for a centralized eviction daemon.
3. GPU-Ready Design: A preliminary architecture showing that cmpxchg-based coordination extends naturally to GPU threads via PCIe AtomicOps, enabling GPU-side filesystem coordination.

4. Experimental Results

The authors evaluated DaxFS using QEMU-emulated CXL 3.0 (with TCP-forwarded atomics) and single-host DRAM-backed DAX.

• Multi-Host Correctness: Under cross-host contention, DaxFS maintained >99% CAS accuracy with no lost updates, validating the lock-free protocol.
• Single-Host Performance (vs. tmpfs):
  • Random Writes: DaxFS achieved 2.68× higher throughput than tmpfs with 4 threads (4,830 MiB/s vs. 1,803 MiB/s) due to the absence of page-cache locks.
  • Random Reads: Achieved 1.18× higher throughput at 64 KB block sizes.
  • Sequential I/O: DaxFS matched or exceeded tmpfs in most scenarios, with significant gains in rewrite scenarios due to lock-free overlay resolution.
• GPU Microbenchmarks:
  • GPU threads performing page cache lookups scaled to 500+ Mops/s (millions of operations per second).
  • Slot-level CAS transitions were limited only by PCIe 5.0 bandwidth (~11.5 Mops/s), not software overhead.
  • Projection: Loading a 140 GB LLM model could be reduced from ~14.5s (conventional two-copy path) to ~2.8s (direct DAX-to-GPU path).

5. Significance

• Paradigm Shift: DaxFS demonstrates that CXL hardware atomics can replace traditional distributed locking and network protocols for shared storage, enabling a "multikernel" style of coordination where independent OS instances share memory directly.
• Efficiency: By eliminating redundant page caches and network overhead, DaxFS maximizes the utility of disaggregated memory pools, crucial for workloads like shared LLM inference and container rootfs sharing.
• Future-Proofing: The design anticipates the integration of accelerators (GPUs) into the storage stack, offering a path for GPU-side filesystem coordination that bypasses the CPU bottleneck entirely.

In summary, DaxFS bridges the gap between CXL hardware capabilities and practical system software, providing a high-performance, lock-free, and scalable filesystem layer for the era of disaggregated memory.