🤖 machine learning

InferScale: GPU-Native KV Injection for Personalized LLM Serving

InferScale is a GPU-native memory system that accelerates personalized LLM serving by precomputing and injecting reusable KV states directly into the vLLM cache, thereby decoupling time-to-first-token latency from the size of retrieved context while maintaining high accuracy.

Original authors: Peter Li, Prashant Pandey

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Peter Li, Prashant Pandey

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot your entire life story so it can answer questions about you. Every time you ask the robot a question, you have to paste your whole biography into its brain before it can think. If your biography is short, the robot thinks fast. But if your biography is long—filled with years of memories, inside jokes, and past adventures—the robot has to re-read and re-process every single word of that biography just to answer a simple "What's my favorite color?" This re-reading process is slow and expensive, like a librarian who has to re-shelve every single book in the library before handing you just one. This is the current state of "personalized" AI: the more memories you give it, the slower it gets to talk to you.

The paper you are about to read tackles this problem using a clever trick from the world of computer science called "Key-Value caching." Think of the robot's brain as a giant whiteboard where it writes down the most important parts of what it has read so far, so it doesn't have to re-read the original text. Usually, if you change the order of the books on the shelf, the robot has to rewrite the whole whiteboard. But what if the robot could just grab the pre-written notes from a drawer and stick them onto the whiteboard instantly, no matter where you wanted them to go? That is the core idea of this research: instead of forcing the robot to re-read your memories every time, we pre-calculate the "notes" for each memory and store them on the robot's super-fast internal hard drive. Then, when you ask a question, we just drop those notes right into the robot's brain.

The Problem: The "Re-Reading" Trap
Today, when you use an AI assistant with a long history of conversations, the system has to find the most relevant memories and paste them into your new question. This is called "prompt injection." Imagine you are asking a friend for advice, but every time you speak, you have to read aloud the last 50 pages of your diary before you can say your actual question. The more pages you read, the longer it takes to get to the advice. Even if you ask the exact same question ten times, the friend has to read those 50 pages ten times. This makes the AI feel sluggish, especially as your personal memory grows.

The Solution: InferScale
The researchers, Peter Li and Prashant Pandey, built a new system called InferScale. Instead of making the AI re-read the memories, InferScale does the heavy lifting once before you even start talking. It takes every single memory fact (like "I love pizza" or "My dog is named Rex") and calculates its "brain notes" (called KV states) ahead of time. These notes are stored directly on the graphics card (the GPU), which is the super-fast brain of the computer.

When you ask a question, the system doesn't send the text of your memories to the AI. Instead, it finds the right "brain notes" from the GPU and injects them directly into the AI's working memory. It's like having a library where, instead of handing you the books, the librarian instantly pastes the exact paragraphs you need onto your desk. The AI never has to re-read the source text; it just uses the notes.

The Magic Tricks
To make this work, the team had to solve two tricky puzzles:

The "Moving Target" Problem (Chunked RoPE):
Usually, these "brain notes" are tied to a specific spot in the text. If you move a memory from the beginning of a story to the middle, the notes become wrong. The researchers invented a technique called Chunked RoPE. Imagine the notes are written on a special kind of paper that can be rotated. No matter where you place the paper on the desk, you can spin it to match the new position perfectly. This allows the system to take a memory stored in one place and drop it anywhere in your conversation without breaking the math.
The "Context" Problem (Context-Window Encoding):
If you just take a single sentence like "He went to the store," the AI might not know who "He" is. If you store the memory in isolation, you lose the context. To fix this, the system stores the memory with a small window of the sentences that came before it (like "John went to the store"). However, it only saves the "brain notes" for the specific memory part, not the whole window. This way, the AI understands the context ("He" is John) without having to re-read the whole paragraph every time.

What They Found
The results are impressive. The researchers tested this on three different AI models (Llama-3.1-8B, Mistral-7B, and Qwen2.5-7B) using a dataset of long conversations.

Speed: In the old way (Mem0), as they increased the number of memories retrieved from 5 to 50, the time it took for the AI to start talking (Time-to-First-Token) jumped by 106% (from 33.2 ms to 68.3 ms). With InferScale, that same increase only caused a 4% slowdown (from 16.6 ms to 17.3 ms).
Throughput: When 100 people used the system at the same time, InferScale handled 3.7 to 4.5 times more questions per second than the old system.
Accuracy: Because the system sometimes stores memories separately, there was a risk the AI would get confused. However, by using the "Context-Window" trick, they recovered almost all the accuracy. At the highest memory load, InferScale got 60.3% of the answers right, which is very close to the old system's 63.3%, but much faster.

Why It Matters
This paper proves that you don't have to choose between a smart, memory-rich AI and a fast one. By moving the memory storage to the GPU and injecting the "brain notes" directly, they decoupled the speed of the AI from the size of your memory. The system requires no retraining of the AI models and no changes to the underlying engine, making it a practical upgrade for the future of personalized AI assistants. Even if the memory data is too big to fit on the fast GPU, they showed that offloading it to the regular computer memory still keeps the system significantly faster than the current methods.

Technical Summary: InferScale

Problem Statement

Large language model (LLM) applications increasingly rely on persistent, personalized context (e.g., accumulated user memories or long conversation histories) shared across multiple requests. Current production memory systems (e.g., Mem0, MemGPT, Zep) address the quadratic prefill cost of long contexts by retrieving a relevant subset of memory facts ( $k$ ) and injecting them as text into the prompt.

However, this "prompt injection" approach incurs a significant serving cost: every retrieved fact must be re-prefilled on every request. Consequently, the Time-to-First-Token (TTFT) grows linearly or quadratically with the retrieval budget ( $k$ ), even when the underlying memory is identical across requests. Existing KV-reuse techniques like prefix caching fail here because they assume reused tokens occupy fixed positions, whereas memory systems dynamically assemble different subsets of facts for each request, violating this assumption.

Methodology

InferScale proposes a GPU-native retrieve-and-inject architecture that replaces repeated prompt prefilling with direct KV injection. Instead of sending retrieved text to the GPU for re-computation, InferScale precomputes the Key-Value (KV) representations of memory facts and injects them directly into the attention cache.

The system is implemented as a plugin for the vLLM serving engine via its KV-connector interface, requiring no model fine-tuning or engine modifications. The architecture consists of two phases:

1. Offline Phase: Context-Window Encoding

To address the loss of cross-fact context when encoding facts independently, InferScale employs Context-Window Encoding.

Process: Each memory fact is encoded together with a configurable window ( $w$ ) of preceding conversation turns.
Optimization: Only the KV tensors corresponding to the target fact are retained; the context tokens are discarded.
Result: The stored KV representation captures the necessary disambiguating context (e.g., resolving pronouns) without requiring the context tokens to be re-prefilled at serving time.

2. Online Phase: Chunked RoPE and KV Injection

To enable the reuse of KV states at arbitrary prompt positions (since retrieved facts are dynamically assembled), InferScale introduces Chunked RoPE.

Pre-RoPE Storage: During offline encoding, keys are intercepted before the application of Rotary Position Embeddings (RoPE). These pre-rotated keys and raw values are stored on the GPU.
Dynamic Rotation: At serving time, retrieved facts are composed into a contiguous segment. The system applies the appropriate RoPE rotation to the pre-stored keys based on their new serving-time positions.
Theoretical Guarantee: The paper proves (Theorem 6) that this re-rotation is mathematically exact under RoPE, ensuring the attention scores are identical to those produced if the facts were prefilled at those positions.
Injection: The composed KV tensors are scatter-copied directly into vLLM's paged KV cache. The model then only prefills the query tokens, attending to the injected memory as if it had been computed.

Key Contributions

Retrieve-and-Inject Inference: The authors demonstrate that retrieved memory need not be re-prefilled. They prove (Theorem 3) that injecting precomputed KV states directly into the attention cache is exactly equivalent to prompt injection for causal decoder transformers, provided the memory is injected at the intended positions.
Chunked RoPE: A novel mechanism for position-independent KV reuse. By storing keys before rotary encoding and re-rotating them at injection time, the system allows facts encoded once to be injected at arbitrary positions without approximation (Theorem 6).
Context-Window Encoding: A technique to recover the accuracy lost by independent fact encoding. By encoding facts with a window of preceding context during the offline phase, the system preserves cross-fact interactions without incurring serving-time recomputation costs.
GPU-Native System Design: InferScale integrates GPU-resident semantic retrieval (using the Jasper index) with reusable KV storage. This keeps both the retrieval index and the KV store on the GPU, eliminating PCIe transfer of retrieved text and repeated prefilling.

Results

Evaluated on the LoCoMo benchmark across three open-weight models (Llama-3.1-8B, Mistral-7B, Qwen2.5-7B) against the state-of-the-art Mem0 system:

Latency (TTFT): InferScale decouples serving latency from the retrieval budget. As $k$ increases from 5 to 50, InferScale's TTFT remains nearly constant (increasing only ~4% for Llama-3.1-8B), whereas Mem0's TTFT increases by 106%. At $k=50$ , InferScale achieves a 3.6–4.8× speedup in TTFT compared to Mem0.
Throughput: Under concurrent load (up to 100 users), InferScale exhibits near-linear scaling, achieving 3.7–4.5× higher throughput than Mem0. Mem0 saturates early due to the computational bottleneck of repeated prefilling.
Accuracy: Independent encoding ( $w=0$ ) initially degrades accuracy. However, with Context-Window Encoding ( $w \ge 20$ ), InferScale recovers nearly all lost accuracy, achieving 60.3% on Llama-3.1-8B at $k=50$ compared to Mem0's 63.3%. The accuracy gap is primarily confined to multi-hop reasoning tasks requiring complex cross-fact attention.
Scalability: The system scales to larger models (Qwen3-14B) with similar latency benefits. The KV store can be offloaded to host DRAM to overcome GPU memory capacity limits, adding only ~3 ms to TTFT while maintaining a 2.3× end-to-end latency advantage over Mem0.

Significance and Claims

The paper claims that InferScale fundamentally changes the serving primitive for personalized LLMs. By moving from token-layer injection to attention-layer injection, it eliminates the quadratic prefill cost associated with retrieved context.

The authors emphasize that:

Exactness: The KV injection mechanism (with Chunked RoPE) is not an approximation but an exact equivalence to prompt injection for the query positions.
Zero Overhead: The system requires no model fine-tuning, no engine modifications, and no changes to existing retrieval pipelines.
Decoupling: It successfully decouples memory-conditioned serving latency from the size of the retrieved context, allowing for high-throughput, low-latency personalized serving even with large retrieval budgets.

The work distinguishes itself from production memory systems (which still pay prefill costs) and existing KV-reuse techniques (which rely on fixed prompt layouts). It positions attention-layer injection as a default primitive for conditioning LLMs on static or semi-static contexts.