Proxics: an efficient programming model for far memory accelerators

This paper proposes "Proxics," an efficient programming model for Near-Data Processing accelerators that adapts familiar OS abstractions like virtual processors and IPC channels into lightweight mechanisms via compilation and interconnect protocols, demonstrating significant performance gains and highlighting the critical need for low-latency CPU-to-accelerator communication.

The Gist

Imagine you are running a massive library (a data center) where the books (data) are stored in a giant, remote warehouse (Far Memory) far away from the main reading room (the CPU).

The Problem:
Usually, when a librarian (the CPU) needs a book, they have to walk all the way to the warehouse, grab it, and bring it back. If the warehouse is huge and the librarian is busy, this walk takes a long time, and the librarian spends more time walking than reading.

To fix this, engineers built Near-Data Processing (NDP) accelerators. Think of these as small, temporary reading desks set up right inside the warehouse. Instead of the main librarian walking back and forth, a small assistant sits at the desk, grabs the books, does the work (like summarizing or sorting), and only sends the final result back to the main room.

The Challenge:
The problem is that these "warehouse assistants" are hard to talk to. Currently, you have to give them very specific, low-level instructions (like "move byte 402 to address 99") using a special, confusing language. It's like trying to hire a new assistant but only being able to communicate via Morse code. It's efficient for the machine, but a nightmare for humans.

The Solution: Proxics
The authors of this paper created Proxics, a new way to talk to these assistants. They decided to use the same "language" we already use for our computers: Processes and Pipes.

Here is how Proxics works, using simple analogies:

1. The "Spawn" (Hiring an Assistant)

Instead of the complex "forking" (copying the whole boss) used in normal computers, Proxics uses a spawn() command.

• Analogy: Imagine you don't clone yourself to go to the warehouse. Instead, you just hand a specific, pre-written instruction manual (a small program) to a new assistant sitting at the warehouse desk. You say, "Here are the rules, go do this specific task." It's lightweight and fast.

2. The "Pipes" (The Messenger Tubes)

This is the most important part. In normal computers, moving data between the main room and the warehouse is slow and clunky. Proxics introduces Pipes.

• Analogy: Imagine a network of pneumatic tubes (like in old banks or hospitals) connecting the main reading room directly to the warehouse desks.
  • Old Way: You write a note, walk to the warehouse, hand it to the assistant, wait, walk back, get the result.
  • Proxics Way: You drop a note into a tube. The assistant grabs it instantly, does the work, and shoots the result back through another tube.
  • Why it matters: The paper shows that if these tubes are slow, the whole system fails. Proxics makes these tubes incredibly fast, allowing the main computer and the warehouse assistants to chat constantly without getting tired.

3. The "Split-Phase" (Multitasking)

The assistants in the warehouse don't have super-fast brains (they are slower than the main CPU), but they are right next to the books.

• Analogy: When an assistant needs to fetch a book, they don't stand there staring at the shelf waiting for it. They shout "I need Book A!" and immediately start working on "Book B" while the conveyor belt brings Book A. By the time they are ready for Book A, it's already there. Proxics teaches the assistants to do this "shout and work" trick automatically.

Why is this a Big Deal?

The researchers built a real prototype using actual hardware (FPGAs) to prove this works. They tested it on three types of tasks:

1. Bulk Operations: Like copying a million books. The assistant does it much faster because they don't have to walk back to the main room.
2. Databases: Imagine a librarian trying to find all books with "Dragon" in the title. The assistant can scan the shelves right there and only send the matching titles back, saving a ton of walking time.
3. Graphs (Social Networks): Imagine tracing connections between people. This is messy and unpredictable. Proxics allows multiple assistants to talk to each other through the tubes to solve the puzzle together, which is something older systems couldn't do well.

The "Gotcha" (The Catch)

The paper also admits a tricky part: Cache Coherence.

• Analogy: Imagine the main librarian and the warehouse assistant both have a copy of the same book. If the librarian updates a page in their copy but forgets to tell the assistant, the assistant might read the old page.
• Because the "warehouse" (Far Memory) doesn't automatically sync with the "main room" (CPU) in current technology, the programmer has to be very careful to tell the system when to "flush" (update) the information. The paper even tested if an AI (LLM) could write code for this and found that even AI struggles with these low-level details!

The Bottom Line

Proxics is a bridge. It takes the powerful, efficient idea of "computing near the data" and wraps it in a familiar, easy-to-use package (Processes and Pipes). It proves that if we build the right "tubes" (communication channels) between our main computers and these new warehouse assistants, we can make data centers faster, cheaper, and much more efficient without needing to be a hardware genius to use them.

Technical Summary

1. Problem Statement

The rise of Far Memory systems (e.g., CXL memory pools) offers scalable capacity and cost benefits but introduces significant performance penalties compared to local DRAM, specifically higher latency (200–400ns vs. ~100ns) and lower bandwidth (20–80 GB/s vs. 100–200 GB/s).

• Current Limitations: Existing approaches like memory tiering (demand paging) rely on the OS to guess access patterns, which is often suboptimal. Near-Data Processing (NDP) offers a solution by moving computation to cores near the memory, but current NDP hardware lacks clean, portable OS abstractions.
• The Gap: Existing programming models for accelerators (like CUDA or pTasks) are either too heavyweight (classical processes) or too specialized (GPU-oriented dataflow). They fail to address the specific constraints of NDP: limited processing power on some accelerators, the need to minimize memory bandwidth usage, and the requirement for fine-grained, low-latency communication between the host CPU and the accelerator.

2. Methodology: The Proxics Model

The authors propose Proxics, a programming model that adapts familiar OS abstractions—Virtual Processes and Inter-Process Communication (IPC) Pipes—for NDP accelerators.

Core Abstractions

1. Channel Programs (CPs) as Processes:

   • Instead of heavy fork() operations, Proxics uses a lightweight spawn() mechanism.
   • Each physical core on the accelerator is a Memory Channel Controller (MCC).
   • An MCC runs a single-threaded Channel Program (CP), which is a compiled binary (1KB–16KB) provided by the application.
   • CPs are compiled specifically for the target hardware, allowing for high efficiency despite limited resources.

2. Split-Phase Execution & Scratchpad Memory:

   • Architecture: MCCs do not rely on complex cache hierarchies. Instead, they use a scratchpad memory (fast, low-latency) and a split-phase copy engine.
   • Mechanism: The copy engine moves data between scratchpad and far DRAM in large units (cache lines/rows). The CP initiates a transfer and continues executing (non-blocking), polling for completion later. This allows the compiler to overlap memory operations with computation (similar to AMAC in databases).

3. Pipe-Based Communication:

   • Proxics implements Unix-like pipes between the Host CPU and MCCs, and between MCCs themselves.
   • Implementation: Unlike traditional pipes that copy data through kernel buffers, Proxics pipes utilize hardware FIFOs and cache-coherent interconnects (or programmed I/O) to achieve ultra-low latency.
   • Granularity: Supports both streaming data and fine-grained synchronization, crucial for irregular workloads.

3. Prototype Implementation

The authors built a complete, functional prototype on the Enzian research computer:

• Hardware: A 48-core ARMv8 ThunderX-1 CPU connected via a cache-coherent interconnect (ECI) to an FPGA.
• FPGA Role: The FPGA hosts 4 MCCs (MicroBlaze soft cores), scratchpad memory, copy engines, and directory controllers. It emulates CXL 1.1/2.0 non-coherent far memory.
• Software: The CPU side runs a standard Linux application with user-space cache management. The MCC side runs compiled Channel Programs using a custom toolchain.

4. Key Results & Evaluation

The evaluation demonstrates that Proxics delivers significant performance gains over CPU-only implementations, even on hardware with modest compute power.

• Bulk Memory Operations:

  • For memset and memcpy on far memory, a single MCC outperforms the CPU for data sizes >8KB.
  • The split-phase copy engine allows the MCC to hide memory latency effectively.

• Random Memory Access (GUPS-like):

  • When the working set exceeds the CPU cache (far memory), the MCC outperforms the CPU by a significant margin, despite the CPU having ~20x higher raw compute power. This highlights the benefit of proximity to memory for latency-bound, irregular workloads.

• In-Memory Databases:

  • Aggregation: For large row sizes (128B+), MCCs outperform CPUs by >39% because the CPU is memory-bound, while the MCC explicitly prefetches data.
  • Filtering: "MCC Streamed" (sending results via pipes) significantly outperforms "MCC Copy" (writing back to DRAM), reducing data movement overhead.

• Unbalanced Tree Search (UTS):

  • Using inter-MCC pipes for dynamic load balancing, a 4-MCC system achieved a 2.3x throughput increase over a 4-core CPU parallel implementation. The pipes allowed efficient distribution of irregular tree branches.

• PageRank:

  • Speedup: Proxics achieved 2.77x speedup on the kron26 graph and 2.40x on twitter compared to a CPU with optimal prefetching.
  • Communication Efficiency: Cache-line based pipes were 4.6x–14.6x faster than register-based pipes.
  • Scalability: The system showed flexible scaling; performance bottlenecks shifted between CPU and MCC depending on the ratio of cores, proving the model's adaptability.

5. Key Contributions

1. Novel Abstraction: Introduced a process-and-pipe model specifically tailored for NDP, bridging the gap between high-level OS concepts and low-level hardware constraints.
2. Efficient Implementation: Demonstrated that lightweight spawn() and split-phase copy engines can achieve near-metal efficiency without complex OS support on the accelerator.
3. Communication Optimization: Proved that low-latency, fine-grained communication (via pipes) is the critical factor for NDP success, often more important than raw compute power.
4. Real Hardware Validation: Unlike many papers relying on simulation, Proxics was implemented and evaluated on a real FPGA-based research platform, providing empirical data on latency, bandwidth, and system overheads.
5. Insights for Future Hardware: Identified that while current CXL hardware is non-coherent, future designs should consider hardware-managed caches or better synchronization mechanisms to reduce software overhead.

6. Significance

• Revitalizing NDP: Proxics provides a practical path to making Near-Data Processing viable for general-purpose applications, moving beyond simple bulk operations to complex, irregular workloads like graph processing and databases.
• Hardware Design Guidance: The results suggest that future far-memory accelerators should prioritize low-latency interconnects and flexible communication channels over raw compute density.
• Software Ecosystem: By using standard abstractions (processes/pipes), Proxics lowers the barrier to entry for developers, potentially enabling a broader ecosystem of applications to leverage far memory acceleration without requiring specialized, vendor-specific SDKs.
• LLM Limitations: The paper notes that current LLMs struggle to correctly implement the necessary cache management and synchronization for non-coherent systems, highlighting the need for higher-level abstractions or compiler assistance in the future.