🤖 machine learning

Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode

This paper introduces cflow, a CPU-first streaming engine co-designed with pipeline-native transformer architectures to overcome memory bandwidth bottlenecks in single-token autoregressive decoding, achieving up to a 2.00x reduction in critical-path weight bandwidth and 5.94 tokens/s on a 32-vCPU server by utilizing L2-sized weight tiling, top-k expert selection, and asynchronous I/O overlap.

Original authors: Tom Poperszky

Published 2026-08-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Tom Poperszky

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, the most powerful computer programs are often described as having a "brain" made of billions of tiny switches. These switches, or weights, are stored in the computer's main memory, like books on a vast library shelf. To make a prediction or generate a single word of text, the computer must fetch the right books from the shelf, read them, and perform a calculation. For years, engineers have assumed that the speed of these calculations was the limiting factor, believing that if they could just make the processor think faster, the program would run faster. However, a new line of research suggests this assumption is wrong for the most common way these programs are used: generating one word at a time. In this specific mode, the processor is not waiting for its brain to think; it is waiting for the memory to deliver the data. The processor is so fast that it spends most of its time sitting idle, staring at an empty shelf, waiting for the next book to arrive. This creates a bottleneck where the speed of the entire system is dictated not by how fast the computer can calculate, but by how fast it can move data from the memory shelf to the processor.

A researcher named Tom Poperszky has tackled this problem by realizing that the standard way of organizing these "books" on the shelf was designed for a different kind of computer entirely. Most modern AI models were built to run on specialized graphics chips, which are excellent at reading many books at once. When these models are run on standard computer processors, the old organization causes the processor to waste time jumping around the shelf, fetching data in a chaotic order that the hardware cannot predict. Poperszky's work proposes a radical solution: instead of trying to force the old models to work better on new hardware, he redesigned both the model and the software that runs it to work together from the ground up. He created a new way to store the model's data in small, neat chunks that fit perfectly into the processor's immediate workspace, and he rewrote the model's internal instructions so that it can read these chunks in a smooth, continuous line without ever stopping to wait.

The core of this new approach is a system called cflow, which acts as a highly efficient librarian for the computer. In a standard setup, when the computer needs to generate a word, it often has to load the entire set of instructions for a specific part of the model, even if it only needs a tiny fraction of them. This is like opening a whole encyclopedia to find a single fact. Poperszky's system changes this by organizing the data into small tiles, roughly the size of a single page, and storing them in the exact order the computer will need them. When the computer asks for the next piece of information, the system slides the next tile into place instantly, keeping the processor busy. Furthermore, for models that use a "mixture of experts"—a technique where the model chooses a few specialized sub-routines for each task—the new system only loads the specific sub-routines needed for that moment, leaving the rest on the shelf. This eliminates the waste of loading thousands of unused instructions for every single word generated.

To make this system work, the researcher had to change the architecture of the model itself. Standard models are built like a strict assembly line where one step must finish completely before the next one begins. This structure forces the computer to wait for the entire line to finish before it can start reading the next set of instructions. Poperszky redesigned the model to allow for a "vertical pipeline," where the computer can start reading the instructions for the next step while it is still finishing the current one. This is only possible because the model was trained to accept a slightly delayed version of the information from the previous step, a change that would cause errors in a standard model but works perfectly in this new design. By training five different versions of these redesigned models, the researcher found that one specific configuration could cut the amount of data the computer needed to move by half. This reduction in data movement directly translates to speed, as the computer spends less time waiting and more time working.

The results of this co-design were measured on real hardware, revealing a clear advantage over existing software. On a standard server computer, the new system generated text at a rate of nearly six words per second, outperforming the best available alternatives which managed only about four and a half words per second on the same machine. This improvement was not just a theoretical calculation; it was a real-world speedup achieved by reducing the number of times the computer's memory had to be accessed. The study also tested several other ideas, such as using explicit instructions to tell the computer where to look for data next. Surprisingly, the tests showed that these instructions did not help and sometimes even slowed the system down, because the computer's own hardware was already doing a better job of guessing what was needed next. This finding is crucial, as it rules out a common optimization technique and confirms that the real gain comes from the fundamental reorganization of the data and the model.

The research also explored how these changes would hold up as models grow larger. The analysis suggests that while the speedup is significant for the models tested, the specific benefits depend on the size of the model and the type of computer used. For very large models, the bottleneck might shift again, but the principle remains: by aligning the model's structure with the way the computer reads memory, it is possible to unlock speeds that were previously thought impossible on standard processors. The work demonstrates that the limitations of current artificial intelligence on everyday computers are not fixed laws of physics, but rather design choices that can be changed. By treating the model and the runtime as a single, unified system rather than two separate pieces, it is possible to create a system that moves data with the efficiency of a well-organized library, allowing the computer to think at the speed of its own hardware. This approach offers a clear path forward for running powerful AI on devices without specialized chips, making advanced intelligence more accessible and responsive in the real world.

Technical Summary: Pipeline-Native Transformers

1. The Problem: The Bandwidth Bottleneck in CPU Inference

The paper argues that single-token autoregressive decoding on CPUs is fundamentally limited by memory bandwidth, not arithmetic compute. While modern CPUs sustain roughly 1 TFLOP/s of compute, they typically offer only ~50 GB/s of main memory bandwidth.

For a single-token decode step, the arithmetic intensity is approximately 4 FLOP/byte (assuming Q4 quantization), which is five times lower than the machine balance of 20 FLOP/byte. Consequently, the CPU spends the majority of its time waiting for weight bytes to stream from RAM rather than performing calculations.

Existing CPU inference runtimes (e.g., llama.cpp, ExLlama2) largely inherit design decisions from GPU-first architectures. These include:

Row-major weight layouts: Optimized for batched access but inefficient for single-token streaming, causing L1-cache misses proportional to weight matrix size.
Standard Mixture-of-Experts (MoE) loading: Loading all expert weights for a layer even if only a subset (top-k) is active, wasting up to 15x bandwidth in models like Gemma 4.
Strict layer dependency: Standard transformers require the complete output of layer $\ell$ before layer $\ell+1$ can begin, preventing vertical pipelining where stages of different layers are processed concurrently.

The paper posits that treating the model architecture and inference runtime as independent entities is a liability in the CPU regime. Instead, a co-design approach is required to reduce the total bytes moved per token.

2. Methodology: Co-Designing Architecture and Runtime

The proposed solution consists of two tightly coupled components: the cflow runtime and a family of pipeline-native transformer architectures.

2.1 The cflow Runtime

cflow is a CPU-first inference engine written in Rust, designed around the CPU memory hierarchy.

Tile-Native Weight Format: Weights are stored as L2-sized tiles (128×256 matrices, ~18 KB at Q4) in compute-consumption order. This ensures that as a tile is processed, the activation vector (residing in L1) is reused across all rows of the tile, minimizing L1-d cache misses.
File Formats:
- .cflow: A per-layer format for standard streaming.
- .vflow: A stage-major format where tiles are grouped by execution stage across layers, enabling vertical pipelining.
Conditional Expert Loading: The runtime uses an expert offset table to load only the top-k selected expert tiles, skipping the rest.
Fused Kernels: It fuses QKV projections and gate+up projections to reduce activation reloads.
Staged Direct-I/O: For disk-resident expert tiers, it uses asynchronous direct I/O to overlap expert fetching with the computation of intervening layers.

2.2 Pipeline-Native Architectures

The paper introduces a formal analysis of the layer dependency Directed Acyclic Graph (DAG) in standard pre-norm transformers. It demonstrates that standard dependencies prevent vertical pipelining. To enable this, the author proposes DAG rewriting via two delay transforms:

Dense Delay ( $\delta_d$ ): The dense FFN at layer $\ell$ consumes the residual stream from layer $\ell - \delta_d$ rather than the current layer's output.
Expert Delay ( $\delta_e$ ): Expert outputs computed at layer $\ell$ are injected into the residual stream at layer $\ell + \delta_e$ .

These transforms break the strict sequential chain, allowing the runtime to fetch weights for layer $\ell+1$ while layer $\ell$ is still computing, provided the dependencies are satisfied by the delayed buffers.

Five candidate architectures were trained on the TinyStories dataset to explore the bandwidth-quality trade-off:

Arch1 & Arch3: Baselines with no delays (1.00× bandwidth reduction).
Arch2_4_combined: Combines $\delta_d=1$ and $\delta_e=2$ . This achieves the maximum bandwidth reduction.
Arch4 (AsyncExperts): Uses $\delta_e=2$ with pre-dense routing (routing before the dense FFN). This achieves the best perplexity but no bandwidth reduction.
Arch5: Uses weight sharing across iterations for parameter efficiency.

3. Key Contributions

The cflow Runtime: A complete, production-quality Rust implementation featuring tile-native formats, fused kernels, and conditional expert loading. It achieves numerical parity with PyTorch references.
Pipeline-Native Taxonomy: A formal definition of dependency-relaxing transforms (dense and expert delays) and five trained architectures demonstrating that these structural changes allow for stable training and convergence.
Delay-Aware Scheduler: A runtime scheduler that constructs a ring-buffered residual history to manage delayed inputs and expert injections, enabling the vertical pipeline schedule.
Empirical Validation: A comprehensive evaluation across eight specific design claims, including negative results where hypotheses were refuted.

4. Results

4.1 Bandwidth and Cache Efficiency

Critical-Path Reduction: The arch2_4_combined architecture achieves a 2.00× reduction in critical-path weight bandwidth (from 9.00 MB/token to 4.50 MB/token) compared to a sequential schedule.
Cache Locality: The tile-streaming layout reduces L1-d read misses by 7.29× compared to a row-major baseline on an Xeon E5-2650 (measured via hardware performance counters).
Conditional Loading: Structural guarantees ensure only top-k expert tiles are read, offering up to a 16× reduction in expert weight bandwidth for large MoE models.

4.2 End-to-End Performance

Throughput: On a 30.9B parameter pipeline-native MoE running on an AWS r6i.8xlarge (Ice Lake, 32 vCPUs), cflow achieves 5.94 tokens/second.
Comparison: This outperforms llama.cpp (4.75 tok/s on a comparable dense 32B model) and the vLLM CPU backend (1.65 tok/s) on the same hardware.
Expert Overlap: On server hardware with NVMe storage, the expert-delay window allows for asynchronous I/O overlap, yielding a net speedup of up to 1.68× over a synchronous blocking approach.

4.3 Negative Results and Refutations

The paper explicitly reports refuted claims to provide a complete picture:

Claim 6 (PREFETCHT0): Explicit prefetching instructions provided no benefit and were sometimes detrimental (4% slower) when the bottleneck was storage-to-RAM bandwidth. The hardware prefetcher already saturates linear RAM access, and explicit hints add cache pressure without overlap.
Claim 8 (Stage-Major Readahead): The theoretical readahead benefit of the .vflow layout was inconclusive at tested scales. The benefit requires asynchronous I/O overlap which was not fully implemented for the non-expert weights in the single-token scheduler; observed variance was attributed to SSD cache states.

5. Significance and Claims

The paper's central thesis is that co-designing the model architecture and inference runtime is necessary to overcome the memory bandwidth bottleneck in CPU inference.

Feasibility: The work proves that rewriting the inter-layer dependency graph of a transformer does not prevent effective learning; the trained architectures converge with competitive perplexity.
Mechanism: The bandwidth reduction is not an approximation but an analytic consequence of the delay parameters and model geometry.
Practicality: The results show that a pipeline-native MoE can outperform state-of-the-art dense models on standard CPU hardware when the runtime is optimized for the specific architecture.

The author concludes that while the "pipeline-native" approach requires training-time architectural changes (making it non-retrofittable to existing checkpoints), the principles of conditional expert loading and tile-streaming weight layouts are immediately applicable to existing runtimes to reduce bandwidth and improve cache locality. The work establishes that the transformer's dependency graph is a design choice, not a fixed constraint, and that exploiting this choice alongside runtime optimization yields optimizations neither could achieve alone.