💬 NLP

MiniPIC: Flexible Position-Independent Caching in <100LOC

MiniPIC is a minimal, flexible vLLM design that enables efficient position-independent caching for recurring structured inputs by storing unrotated K vectors and applying per-request positional encodings during attention, thereby achieving significant throughput and latency improvements with fewer than 100 lines of core-engine changes.

Original authors: Nathan Ordonez (IBM Research), Thomas Parnell (IBM Research)

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nathan Ordonez (IBM Research), Thomas Parnell (IBM Research)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-speed library where a librarian (the AI) has to read thousands of books to answer a single question. In modern AI, this "reading" phase is called prefill.

Usually, if two people ask similar questions, the librarian can reuse parts of the books they've already read. However, current library systems (like vLLM) are very rigid: they can only reuse a page if it appears in the exact same spot in the book. If you move a paragraph from page 10 to page 100, the librarian treats it as a brand-new page and has to read it all over again, even though they just read it a second ago.

MiniPIC is a clever, tiny upgrade to this library system that solves this problem without rebuilding the whole building. Here is how it works, using simple analogies:

1. The Problem: The "Sticky Note" Dilemma

In current AI systems, the "memory" of what was read (called the KV cache) has a sticky note attached to every page saying, "I am on page 10."

The Conflict: If you want to use that same page of text for a new request where it belongs on page 100, the library system gets confused. It can't have the same physical page be both "Page 10" and "Page 100" at the same time for different people.
The Old Fix: Previous solutions tried to make photocopies of the page, move them around, and re-label them. This is slow, wastes space, and creates traffic jams in the library.

2. The MiniPIC Solution: The "Blank Page" Strategy

MiniPIC changes the rules of the library in two simple ways:

A. The "Unlabeled" Shelf (Position-Free Cache)
Instead of writing "Page 10" on the physical page, MiniPIC keeps the pages blank regarding their location. The page just contains the words.

How it works: When the librarian reads the page, they only decide where it is at that exact moment. If they are reading for Person A, the page is "Page 10." If they are reading for Person B, the same physical page instantly becomes "Page 100."
The Benefit: You don't need to photocopy or move pages. The same physical memory block can serve multiple people at different positions simultaneously.

B. The "Magic Tokens" (User-Controlled Primitives)
Since the librarian no longer knows where a page belongs just by looking at it, the user (or a manager) gives the librarian three special "magic tokens" (special words) to tell the system how to handle the books:

Padding: "Fill the empty space so this book ends neatly on a shelf boundary."
SSEP (Span Separator): "This section is independent. You can reuse this chunk of text even if it's in a different book or a different spot." (This breaks the rigid "must match the start" rule).
PDEP (Prompt Depend): "This part depends on everything that came before it. Do not reuse this; read it fresh."

By using these three tokens, the user can tell the system exactly which parts of the text are reusable "chunks" (like documents or code files) and which parts are unique.

3. The "Conveyor Belt" Scheduling

The paper also introduces a smarter way to organize the work.

Old Way: The library would finish reading all the reusable chunks for everyone, put a "STOP" sign on the line, and then start answering the actual questions. This leaves the librarian sitting idle.
MiniPIC Way (Interleaved Scheduling): The librarian starts reading the reusable chunks for the next person while still answering the current person's question. It's like a chef chopping vegetables for the next dish while the current dish is still simmering. This keeps the kitchen (the GPU) busy 100% of the time.

The Results: Fast, Small, and Flexible

The paper claims that by making these changes, they achieved:

Speed: A 49% increase in how fast the system can read and prepare data (prefill throughput) compared to the standard system.
Instant Access: For cached documents, the time to get the first answer dropped by up to 100 times (two orders of magnitude).
Tiny Footprint: The entire system required changing only 78 lines of code in the core engine (plus a custom "attention" tool). It's like fixing a car engine by swapping out two tiny spark plugs rather than rebuilding the whole engine.
No Penalty: When the system doesn't use these tricks, it only slows down by about 5.7%, which is barely noticeable.

Summary

MiniPIC is a lightweight, flexible upgrade that lets AI systems reuse text chunks regardless of where they appear in a prompt. It does this by removing "location labels" from the memory and letting users mark which parts are reusable with simple special words. This eliminates the need for complex, slow copying operations and allows the AI to work much faster, especially when dealing with long documents or repeated information.

Technical Summary: MiniPIC: Flexible Position-Independent Caching in <100LOC

1. Problem Statement

Modern Large Language Model (LLM) inference is increasingly dominated by long-context prefill workloads, particularly in Retrieval-Augmented Generation (RAG) and agentic scenarios. These workloads involve repeatedly prefilling predictable, structured input segments (termed "spans"), such as documents, tool descriptions, or memory states.

Current production inference engines like vLLM and SGLang utilize prefix caching, which hashes fixed-size KV-cache blocks based on their preceding tokens. While effective for identical initial prompts, this approach fails to reuse KV entries for spans that appear at different absolute positions or after different prefixes (e.g., the same document appearing after different system prompts).

Existing Position-Independent Caching (PIC) solutions face a fundamental systems conflict regarding positional encodings:

The Conflict: Most LLMs use Rotary Position Embeddings (RoPE). Standard implementations store keys in the KV cache after applying RoPE. If a span is cached at offset 0 and reused at offset 2048, the cached keys carry the wrong rotation.
The Concurrency Barrier: In a serving system, physical KV blocks are shared across concurrent requests. If the same span is needed simultaneously at two different offsets, a single physical block cannot hold two different positional rotations.
Prior Art Limitations:
- External Caching: Storing spans outside the GPU (e.g., in host memory) incurs significant host-to-device transfer overhead.
- Complex Internal Solutions: Implementations like SPNL and MEPIC attempt to solve this within the engine but require substantial code changes, including complex block repositioning engines (e.g., SPNL's CIDRA algorithm for cycle detection and block copying), new memory management subsystems, and dedicated eviction policies. These introduce memory amplification, copy cycles, and unfused operations that degrade performance.

The paper posits that flexible PIC requires a solution that removes positional conflicts from the shared cache while keeping engine modifications minimal and aligned with existing architecture.

2. Methodology: MiniPIC

The authors present MiniPIC (Minimalistic PIC), a design built on two core ingredients: a positional-encoding-free KV cache and user-controlled cache-reuse primitives.

2.1 Position-Free KV Implementation

Instead of storing pre-rotated keys, MiniPIC stores unrotated key vectors ( $\tilde{k}_j$ ) in the KV cache.

Mechanism: The Rotary Position Embedding (RoPE) is applied on-the-fly inside the attention kernel using per-request logical positions.
Mathematical Formulation: The attention score is computed as:
$\text{score}(t, j) = \frac{(R_t q_t)^\top (R_{\pi(j)} \tilde{k}_j)}{\sqrt{d}}$
Where $R$ is the rotation matrix, $q_t$ is the query, and $\pi(j)$ is the logical position of the cached key $j$ in the current request.
Benefit: This decouples the physical KV block from a specific position. The same block can be referenced by multiple concurrent requests at different offsets without requiring duplication or re-rotation of shared memory.
Implementation: This requires modifying the attention kernel to perform a cosine/sine table lookup and fused multiply-adds per key element. It is algebraically equivalent to re-rotating cache hits but avoids mutating shared blocks.

2.2 User-Controlled Cache Reuse Primitives

To manage span composition without complex engine-side canonicalization, MiniPIC exposes three user-facing primitives implemented as special tokens. These modify the engine's block-hashing behavior:

Padding: Users pad spans to KV-block boundaries to ensure special tokens appear at the start of blocks.
SSEP (Span Separator): When placed at the start of a block, SSEP resets the hash chain ( $h_i = H(h_0, x_i)$ ). This makes the span independently cacheable regardless of its preceding context, enabling Block-Attention and EPIC styles.
PDEP (Prompt Depend): When placed at the start of a block, PDEP forces the hash to depend on the entire preceding prompt ( $h_i = H(h_0, x_{\le i})$ ). This prevents false cache hits for suffix blocks that genuinely depend on prior context.

Scheduling: The paper introduces Interleaved Span-Prefill Scheduling (ISPS). Instead of batching all span prefills followed by a barrier and then final prompts, ISPS pipelines span prefills with ongoing decode steps. This keeps the GPU saturated and eliminates idle time, leveraging vLLM's continuous batching.

3. Key Contributions

Systems Analysis of Positional Conflicts: A detailed study demonstrating that retaining positional encodings in shared KV caches leads to cascading problems (memory races, amplification, copy cycles) when attempting PIC.
Minimal, Flexible Implementation: A position-free KV implementation that stores unrotated keys and applies RoPE in the attention kernel. Excluding the modular attention backend, it requires only 78 lines of code (LOC) changes to the vLLM core (61 LOC of new functionality).
User-Controlled Interface: Demonstration that three special tokens (Padding, SSEP, PDEP) are sufficient to express multiple span techniques (Block-Attention, EPIC, Prompt Cache) without requiring engine-managed chunk canonicalization or mid-prompt recomputation policies.
PIC-Aware Interleaved Scheduling: A scheduling policy that overlaps span prefills with final-prompt requests, achieving significant throughput improvements over barrier-separated baselines.
Native Compatibility: The design natively integrates with vLLM's CPU offload implementations, as unrotated blocks can be moved between GPU and host without tracking or adjusting positions.

4. Experimental Results

Experiments were conducted on a single NVIDIA H100 GPU using the 2WikiMultihopQA dataset (12,576 samples) and the Tulu3-block-ft model (Llama-3-8B architecture).

Throughput: MiniPIC with ISPS achieved 48.01 samples/s, representing a 49% improvement over baseline vLLM (32.21 samples/s) and a 33% improvement over SPNL (36.14 samples/s).
Overhead: The "PIC disabled" configuration (position-free keys but no span signaling) incurred only a 5.7% overhead compared to baseline, confirming the minimal cost of the RoPE-in-kernel modification.
Latency (TTFT): MiniPIC reduced the time-to-first-token for cached spans by up to two orders of magnitude compared to baseline vLLM and significantly outperformed SPNL by eliminating the memory-copy overhead of CIDRA.
Scalability: The system preserved linear prefill scaling for uncached spans.
Code Footprint: The implementation required only 78 core LOC changes (plus 251 LOC in the modular Triton attention backend), contrasting sharply with prior works that introduce entire new subsystems.

5. Significance and Claims

The paper argues that flexible Position-Independent Caching does not require a large, engine-managed PIC stack. The authors claim that two ingredients are sufficient:

Position-free keys in the KV cache to eliminate concurrency conflicts and the need for complex block repositioning (like CIDRA).
A minimal user-controlled interface (via special tokens) to manage cache reuse, shifting the responsibility of span management to the user or orchestration layer.

By removing positional encodings from shared memory, MiniPIC avoids the structural consequences of memory amplification and copy cycles inherent in previous approaches. The result is a system that is simpler (fewer lines of code), faster (no copy/reposition overhead), and compatible with existing production features like CPU offload. The paper concludes that for RoPE-based span-aware serving, the simplest and most effective design is to keep the cache position-free and the interface user-controlled.