🤖 machine learning

CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIM

This paper proposes CHIME, the first Attention-FC Disaggregated inference system leveraging DIMM-PIM accelerators with bubble-free pipelining and hybrid-grained re-layout to balance memory capacity and bandwidth constraints, achieving up to 5.15× speedup over state-of-the-art HBM-PIM solutions.

Original authors: Qingyuan Liu, Liyan Chen, Haocheng Wang, Yanning Yang, Dong Du, Zhigang Mao, Naifeng Jing, Yubin Xia, Haibo Chen

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qingyuan Liu, Liyan Chen, Haocheng Wang, Yanning Yang, Dong Du, Zhigang Mao, Naifeng Jing, Yubin Xia, Haibo Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to bake the world's biggest, most complex cake. You have a super-fast, high-tech oven (the GPU) that can mix ingredients and bake layers in a flash, but it's tiny and can only hold a few trays at once. Then, you have a massive, slow-moving pantry (the memory) where you keep all the flour, sugar, and eggs. If you try to bake a cake that requires a mountain of ingredients, your oven runs out of space to hold the trays, or it spends all its time waiting for the pantry to hand over the next batch of flour. This is exactly the problem facing modern "Large Language Models" (LLMs)—the super-smart AI chatbots that write code, tell stories, and solve math problems. As these models try to understand longer and longer conversations (like reading a whole novel in one go), they get stuck waiting for data to move from the pantry to the oven.

To fix this, scientists came up with a clever idea called "disaggregation." Instead of forcing the oven and the pantry to live in the same tiny box, they split the work. They let the oven handle the heavy lifting of mixing and baking (the math), while sending the massive pile of ingredients to a separate, giant pantry that has special helpers (called PIM, or Processing-in-Memory) to sort and fetch them quickly. The hope was that by splitting the team, the whole operation would speed up. But here's the twist: just giving the pantry more helpers or faster shelves didn't always make the cake bake faster. Sometimes, the pantry got so big it ran out of room, and other times, it was so fast that the oven just sat there waiting, bored.

This is where a team of researchers from Shanghai Jiao Tong University stepped in with a new system called CHIME. They realized that the old way of splitting the work was missing a crucial rule: you can't just make one part of the team super strong; you have to make sure the weakest link isn't holding everyone back. They discovered that in these split systems, the speed is limited by whichever resource is scarcer—either the pantry's storage space or how fast it can hand things over. If you have a pantry with infinite speed but only a cup of flour, you're stuck. If you have a warehouse full of flour but a snail-paced delivery truck, you're also stuck.

The paper proposes CHIME as a solution that balances both. Instead of using expensive, tiny, super-fast pantries (like the ones currently attached to high-end graphics cards), CHIME uses standard, massive computer memory sticks (DIMMs) that have been upgraded with little helpers right inside them. This gives the system a huge amount of storage and a good amount of speed, without breaking the bank. But simply plugging these new memory sticks in wasn't enough; the researchers had to invent a new way to organize the data so the helpers didn't trip over each other, and a new scheduling system to make sure the oven and the pantry are always working at the same time, never waiting for the other to catch up.

Through detailed computer simulations, the authors found that this new approach could make the AI up to 5.15 times faster than the current best methods using those tiny, expensive pantries. They also showed that simply making the pantry faster or bigger on its own doesn't work; you have to grow both at the same time to see real gains. In short, CHIME is a smarter, more balanced way to run these giant AI brains, ensuring that no part of the team is left standing around doing nothing while the rest of the kitchen is in chaos.

Technical Summary: CHIME - A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIM

1. Problem Statement

The demand for Large Language Models (LLMs) to process long sequences has created a significant performance bottleneck in the decoding phase, driven by the increasing memory capacity and bandwidth requirements of the KV cache. While Attention-FC Disaggregated (AFD) systems have emerged as a solution—offloading memory-bound Attention operations to memory-rich accelerators (e.g., CPUs, HBM-PIM) while retaining compute-bound Fully-Connected (FC) operations on GPUs—existing implementations face critical inefficiencies.

The authors identify a counter-intuitive phenomenon: simply enhancing the accelerator (e.g., increasing bandwidth) does not guarantee improved system throughput. Prior AFD systems often fail to balance memory capacity and bandwidth, leading to resource underutilization or constrained throughput. This issue is explained by a "Liebig's Law" of systems, where the overall throughput is limited by the scarcest resource (either the accelerator's memory bandwidth or its capacity), regardless of how abundant the other resource may be.

2. Methodology

A. Disaggregated Roofline Model (DRM)

To characterize AFD performance, the authors propose the Disaggregated Roofline Model (DRM). Unlike traditional models that analyze devices in isolation, DRM holistically characterizes the interdependencies between the GPU (executing FC) and the accelerator (executing Attention).

Key Insight: The system throughput is determined by the lower of the two device throughputs.
Bottleneck Analysis:
- If the accelerator is the bottleneck, throughput is limited by its memory bandwidth.
- If the GPU is the bottleneck, throughput is limited by the accelerator's memory capacity (which restricts the maximum batch size).
Conclusion: Effective AFD design requires a balanced trade-off between capacity and bandwidth. Scaling only one dimension yields diminishing returns once the other becomes the limiting factor.

B. The CHIME System Proposal

Based on DRM analysis, the authors propose CHIME, the first AFD system integrating DIMM-PIM (Processing-in-Memory integrated with standard DIMM modules) as the accelerator. DIMM-PIM is chosen because it offers a more balanced configuration than HBM-PIM (which suffers from capacity limits) or CPU offloading (which suffers from bandwidth limits).

CHIME consists of two main components:

1. CHIME-PIM (Hardware Design)

A novel DIMM-PIM hardware architecture designed for efficient attention computation, addressing synchronization overheads inherent in distributed DRAM chips:

Bubble-Free Pipelining: To eliminate data synchronization overheads during cross-chip transfers (e.g., for Softmax accumulation), CHIME employs a pipelined execution that overlaps communication with computation. By decoupling internal and external memory buses and using specific head-mapping methods, the system ensures that data transfer time ( $T_{comm}$ ) is less than or equal to computation time ( $T_{comp}$ ), preventing pipeline bubbles.
Hybrid-Grained Re-layout: To address layout mismatches where a single data element might span multiple chips (preventing PIM execution), CHIME utilizes a re-layout unit on the rank PU (Buffer Chip). This performs fine-grained re-layout (arranging bits of a single element contiguously on one chip) and coarse-grained re-layout (mapping heads to specific numbers of chips) during data offloading/onloading, minimizing latency.

2. CHIME-sys (System Software & Scheduling)

A coordinated inference system that minimizes cross-device synchronization overheads:

Rankset-Granular Communication-Computation Overlapping: Instead of treating all memory ranks as a single unit, CHIME defines a "rankset" (one rank from each channel) as the finest granularity for independent operations. This allows communication on one rankset to occur while other ranksets perform computation, effectively hiding PCIe transfer overheads.
Alignment-Predicting Scheduling: To prevent idle "bubbles" caused by imbalanced completion times between the GPU and PIM, the scheduler uses performance modeling (Random Forest Regression for GPU, linear models for PIM) to predict execution latencies. It dynamically selects and groups requests into sub-batches such that the predicted latencies on both devices are aligned, maximizing parallel execution.

3. Key Contributions

Disaggregated Roofline Model (DRM): A new performance model that reveals the "Liebig's Law" governing AFD systems, demonstrating that throughput is constrained by the weaker of the two resources (bandwidth or capacity).
CHIME Architecture: The first AFD system to leverage DIMM-PIM, offering a balanced solution that alleviates the capacity bottlenecks of HBM-PIM and the bandwidth bottlenecks of CPU offloading.
Hardware Innovations: Introduction of bubble-free pipelining and hybrid-grained re-layout to eliminate synchronization stalls within the DIMM-PIM hardware.
System Innovations: Development of rankset-granular overlapping and alignment-predicting scheduling to maximize cross-device resource utilization and minimize idle time.

4. Experimental Results

The authors evaluated CHIME using three LLM models (OPT-66B, QWEN-72B, GPT-175B) and three real-world traces (OpenR1, Dolphin-r1, OpenThoughts) on a simulated DGX-A100 platform.

Throughput: CHIME achieves up to 5.15× speedup over state-of-the-art HBM-PIM solutions and up to 7.21× over rank-level DIMM-PIM (R-PIM) baselines.
Batch Size: CHIME supports significantly larger batch sizes (up to 6.6× larger than baselines) due to the 2TB capacity of host memory, which directly drives the throughput improvement.
Scalability: The results confirm the DRM analysis: scaling only bandwidth or only capacity yields diminishing returns. Simultaneous scaling of both (as achieved by CHIME) is required for optimal throughput.
Efficiency: CHIME reduces total energy consumption by 40% compared to HBM-PIM baselines, primarily by enabling larger batch sizes on the GPU and reducing weight loading frequency.
Overhead Reduction: The proposed optimizations reduce PCIe communication overhead by up to 75% and eliminate idle bubbles in the pipeline.

5. Significance and Claims

The paper claims that CHIME establishes a new paradigm for efficient long-context LLM inference by shifting the focus from simply adding high-bandwidth memory (like HBM) to achieving a balanced trade-off between capacity and bandwidth.

The authors assert that:

Liebig's Law is a fundamental constraint for AFD systems that prior works have overlooked, leading to inefficient resource provisioning.
DIMM-PIM represents a superior accelerator choice for AFD systems due to its scalability, configurability, and economic efficiency (lower cost per GB compared to HBM).
The hardware-software co-design in CHIME is essential to overcoming the synchronization challenges of distributed DRAM chips, enabling the theoretical benefits of PIM to be realized in practice.

The paper concludes that by adhering to the principles derived from the DRM and leveraging the specific advantages of DIMM-PIM, CHIME achieves significantly higher throughput and resource efficiency, making it a viable solution for the growing demands of long-context LLM applications.