🤖 machine learning

Efficient, VRAM-Constrained xLM Inference on Clients

This paper introduces pipelined sharding, a novel CPU-GPU hybrid scheduling technique that significantly improves inference speed and reduces VRAM requirements for large language and vision-language models on client systems, achieving up to 30x throughput gains and 10x VRAM reduction compared to aggressive baselines.

Original authors: Aditya Ukarande, Deep Shekhar, Marc Blackstein, Ram Rangan

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aditya Ukarande, Deep Shekhar, Marc Blackstein, Ram Rangan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Tiny Suitcase" Dilemma

Imagine you have a massive, incredibly smart library (a Large Language Model or AI) that you want to carry with you in your backpack. The problem is that your backpack (your computer's VRAM or video memory) is very small.

If you try to stuff the whole library into the backpack, it won't fit. If you try to leave the library at home and only bring a few pages, the AI becomes slow or stupid because it has to run back and forth to the house to get more information.

Current solutions often force you to throw away parts of the library (making the AI less accurate) or require you to have a giant, expensive backpack that most people don't own.

The Solution: "Pipelined Sharding" (The Smart Moving Crew)

The authors, working for NVIDIA, created a new system called Pipelined Sharding. Think of this as a highly organized, super-smart moving crew that knows exactly how to pack and move your library between your house (the CPU or main memory) and your backpack (the GPU or video memory) without you having to lift a finger.

Here is how it works, broken down into three simple steps:

1. The "Test Drive" (Profiling)

Before the moving crew starts, they run a quick "test drive" on your specific computer. They check:

How fast is your CPU?
How wide is the hallway between your house and backpack (the PCIe connection)?
How much space is actually in your backpack?

They don't guess; they measure. This creates a "profile" of your computer's unique strengths and weaknesses.

2. The "Three-Plan" Strategy (Planning)

Based on the test drive, the system prepares three different moving strategies for every possible situation:

Plan A (The Sprinter): If you have a huge backpack and a fast hallway, the crew puts everything in the backpack and runs fast.
Plan B (The Relay): If the backpack is small but you have many strong helpers (CPU threads), the crew splits the work. Some books stay in the house and are read by the helpers, while others are in the backpack. They pass the books back and forth efficiently.
Plan C (The Overlap): If the hallway is wide, the crew starts carrying the next batch of books while the current batch is being read. This hides the time it takes to move things.

The system doesn't just pick one plan forever. It looks at what you are doing right now. Are you reading a short sentence (interactive mode) or a whole chapter (batch mode)? It picks the best plan for that exact moment.

3. The "Sub-Layer" Packing (Sharding)

Instead of moving whole chapters at once, this crew breaks the library down into tiny sections (sub-layers).

The VIP Section: The most important parts (like the "Attention" mechanism, which helps the AI focus on what matters) are always kept in the backpack because they are needed constantly.
The Storage Section: The less critical parts stay in the house and are only brought out when absolutely necessary.

This "sub-layer" approach allows the system to fit massive models into tiny backpacks that previously couldn't hold them at all.

The Vision Upgrade: "VLMOpt" (The Camera Lens)

The paper also tackles Vision Language Models (VLMs), which are AIs that can "see" pictures.

The Problem: High-resolution photos are like giant, heavy paintings. Trying to load a 4K image into a small backpack crashes the system.
The Fix: They added a special trick called VLMOpt.
1. Offloading: They keep the heavy "painting tools" (weights) in the house and only bring the specific brush strokes needed for the current moment.
2. Flash Attention: They use a new way of looking at the picture that doesn't require remembering every single pixel at once, saving massive amounts of space.
3. No Overlap: They ensure the "painting" part and the "reading" part of the AI don't try to sit in the backpack at the same time. They take turns, so the backpack never gets too full.

The Results: What Actually Happened?

The paper tested this on real computers (from laptops to high-end desktops) with various AI models. Here is what they found, sticking strictly to their claims:

Speed: For interactive use (like chatting with an AI), the system made the AI 6.7 times faster to start talking and 30 times faster at generating words compared to previous methods.
Fitting the Unfit: They were able to run a massive 77GB AI model on a computer with only 2GB of video memory. That's like fitting a 77-story building into a shoebox.
Batching: When processing many requests at once (batch mode), the system was up to 8.2 times faster.
Gaming: When running an AI alongside a video game (like Cyberpunk 2077), the system found a "sweet spot" where both the game and the AI ran smoothly without crashing each other.
Vision: For the "Cosmos-Reason1" AI (which looks at images), they reduced the memory needed by 10 times, allowing high-resolution image analysis on devices that previously couldn't handle it.

The Bottom Line

This paper introduces a "smart scheduler" that acts like a master conductor for your computer's resources. It doesn't make the AI less accurate (it's "lossless"); instead, it figures out the most efficient way to shuffle data between your main memory and your video memory.

By doing this, it allows developers to run huge, smart AIs on regular laptops and gaming PCs, even if those computers have very limited video memory. It turns a "too big to fit" problem into a "just right" solution by constantly adapting to the computer's current conditions.

1. Problem Statement

The paper addresses the critical challenge of running high-accuracy, large-scale AI models (collectively termed xLMs, including Dense LLMs, Mixture-of-Experts (MoE) LLMs, and Vision-Language Models (VLMs)) on client-side systems (desktops and laptops) with limited Video RAM (VRAM).

The Constraint: Client GPUs often have significantly less VRAM than the total memory required by modern high-accuracy models (e.g., a 77GB model on a 16GB GPU).
The Limitation of Existing Solutions:
- Quantization/Pruning: These techniques reduce model size but sacrifice accuracy, which is unacceptable for high-fidelity applications.
- Static Offloading: Existing hybrid scheduling (e.g., in llama.cpp) often relies on manual configuration (e.g., specifying the number of GPU layers) and fails to dynamically adapt to varying system conditions (CPU threads, PCIe bandwidth) or inference phases (prefill vs. decode).
- Lack of Universality: No single existing product efficiently handles interactive modes, batch processing, high-resolution VLMs, and both dense/MoE architectures simultaneously under strict VRAM budgets.

2. Methodology

The authors propose a novel, benchmark-profile-guided hybrid scheduling technique called Pipelined Sharding, combined with a set of vision-specific optimizations called VLMOpt.

A. Pipelined Sharding (Core Innovation)

This is a fully automatic scheduling technique that breaks the inference graph at the sub-layer level (e.g., splitting Attention and FFN blocks) rather than just at the layer level. It operates in three phases:

Install Phase (Profiling):
- A benchmark suite runs on the specific client hardware to profile kernel performance (FLOPS) for various tensor sizes, quantization levels, and CPU thread counts.
- Crucially, it profiles under concurrent load (simulating PCIe traffic and memory controller contention) to generate accurate timing estimates for hybrid CPU-GPU execution.
- This creates a small profile database (~170KB) used for cost estimation.
Planning Phase (Schedule Generation):
- The system defines a set of "Token Tiers" (e.g., 1, 4, 16, 64 tokens per iteration) representing different inference loads.
- For each tier, it generates three distinct scheduling plans based on the current VRAM budget and system conditions:
  - GPU-Only: All unpinned sub-layers run on the GPU; weights are streamed via PCIe. Optimal for high compute loads (large batches/contexts) where PCIe latency is hidden by compute.
  - Static: High-priority sub-layers (Attention, KV Cache) are pinned to VRAM; low-priority ones (FFN) run on the CPU. Minimizes PCIe traffic but relies on CPU compute.
  - Dynamic: A hybrid approach where weights for GPU sub-layers are streamed from sysRAM while the CPU executes previous sub-layers. This overlaps PCIe transfers with compute.
- A Roofline-based timing estimator uses the profile database to predict the latency of each plan and selects the optimal one for the specific token tier.
Inference Phase (Execution):
- During inference, the system determines the "new token" count for the current batch.
- It looks up the pre-calculated optimal schedule for that token tier and executes it.
- This allows the system to adapt seamlessly between interactive mode (Batch Size = 1, low tokens) and batched mode (Batch Size > 1, high tokens).

B. VLMOpt (Vision-Language Optimization)

To support high-resolution VLMs (specifically the Cosmos-Reason1 model), the authors implemented three optimizations within llama.cpp:

Vision Tensor CPU Offloading: Vision model weights are pinned to CPU memory and streamed to the GPU only when needed, reducing VRAM footprint.
Flash Attention for Vision: Implements Flash Attention in the vision encoder to reduce the $O(N^2)$ memory complexity of attention scores for high-resolution images.
VRAM Overlap Avoidance: Restructures execution to ensure vision encoding completes and frees VRAM before language decoding begins, ensuring peak VRAM usage is the maximum of the two parts rather than their sum.

3. Key Contributions

Pipelined Sharding Algorithm: A novel, automatic hybrid scheduler that adapts to system conditions (CPU threads, PCIe bandwidth) and inference states (context vs. decode) without manual tuning.
VLMOpt Implementation: A suite of optimizations enabling high-resolution VLM inference on client hardware with drastically reduced VRAM requirements.
Open-Source C++ Implementation: The solution is implemented in llama.cpp (a C++-first framework), making it accessible to game developers and physical AI engineers.
Comprehensive Evaluation: Rigorous testing across Dense LLMs, MoE LLMs, and VLMs on diverse client hardware (laptops to high-end desktops).

4. Results

The evaluation demonstrates significant performance improvements over aggressive baselines (manually tuned llama.cpp and vLLM):

Interactive Mode (Batch Size = 1):
- TTFT (Time-to-First-Token): Improved by up to 6.7×.
- TPS (Tokens Per Second): Improved by up to 30× (e.g., for the 77GB Qwen235B model).
- Viability: Enables massive models (e.g., 77GB) to run interactively on GPUs with as little as 2GB VRAM (achieving >5 TPS, the threshold for human reading speed).
Batched Mode (Batch Size > 1):
- Throughput improvements of up to 8.2× compared to baselines.
- The scheduler effectively shifts work to the GPU as batch size increases, avoiding CPU bottlenecks.
VLM Performance (Cosmos-Reason1):
- VRAM Reduction: Reduced VRAM demand by 10× (from ~20GB to ~2GB), enabling high-resolution (1440p) inference on client systems where it was previously impossible.
- Latency: Achieved up to 1.78× speedup in End-to-End Latency (E2EL) compared to baselines at equivalent VRAM budgets.
Co-Execution: In scenarios running a game (Cyberpunk 2077) and an LLM concurrently, the solution allows the game to retain more assets in VRAM, improving both game FPS and LLM TPS by avoiding resource contention.

5. Significance

This work bridges the gap between the high accuracy requirements of modern AI and the hardware constraints of consumer client devices.

Lossless Inference: It proves that high-accuracy models can run on client hardware without sacrificing model fidelity through quantization or pruning.
Democratization: By enabling high-resolution VLMs and large MoE models on standard laptops and desktops, it empowers developers to build advanced "Physical AI" and in-game AI assistants without relying on expensive cloud infrastructure.
Adaptability: The token-tier and profile-driven approach provides a robust framework for future hardware evolution, ensuring that inference performance scales with available resources (CPU threads, PCIe bandwidth) rather than being statically bound to a fixed configuration.

In summary, the paper presents a production-ready, automated solution that makes the next generation of client-side AI feasible, efficient, and lossless.