💬 NLP

FastOCR: Dynamic Visual Fixation via KV Cache Pruning for Efficient Document Parsing

FastOCR is a training-free framework that accelerates document parsing in Vision-Language Models by leveraging the temporally sparse nature of visual attention to dynamically prune and reuse KV cache tokens at each decoding step, achieving significant latency reduction with minimal accuracy loss.

Original authors: Zihan Tang, Leqi Shen, Hui Chen, Ao Wang, Ben Wan, Yan Feng, Ke Zhang, Sicheng Zhao, Tongxuan Liu, Guiguang Ding

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zihan Tang, Leqi Shen, Hui Chen, Ao Wang, Ben Wan, Yan Feng, Ke Zhang, Sicheng Zhao, Tongxuan Liu, Guiguang Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to read a massive, dense textbook page using a super-smart robot assistant. The problem is that the robot is trying to look at every single word, letter, and speck of dust on the entire page at the exact same time, all while it tries to type out the next letter of the sentence.

This is like trying to drink from a firehose. The robot gets overwhelmed, it's incredibly slow, and it burns a lot of energy just to process the sheer volume of information, even though it only needs to read one word at a time.

This paper, FastOCR, introduces a new way for these robots to read documents that is much faster and smarter. Here is how it works, broken down into simple concepts:

The Problem: The "Firehose" Approach

Current AI models (called Vision-Language Models) are great at reading text from images. But when they look at a document, they treat the whole page as a giant pile of data. They try to keep every single "visual token" (a digital piece of the image) in their short-term memory.

The Old Way (Physical Eviction): Some previous methods tried to speed things up by permanently throwing away parts of the image they thought were unimportant.
Why that fails for documents: Imagine you are reading a page of text and you decide to throw away the top half of the page because "it looks like a header." If the text you need is actually in that header, or if the layout is tricky, you lose the information forever. In document reading, every single pixel matters. Throwing anything away is like tearing pages out of a book; you can't put them back, and the story becomes broken.

The Solution: The "Human Reader" Approach

The authors noticed something fascinating: Humans don't read like robots.
When you read a page, you don't stare at the whole page at once. Your eyes (your "fixation") land on one word, then the next, then the next. You only focus intensely on a tiny spot at any given moment, but your brain knows the rest of the page is still there if you need to look back.

FastOCR mimics this human behavior. It doesn't throw anything away; it just changes what the robot looks at right now.

How FastOCR Works (The Two Magic Tricks)

The system uses two main tricks to make the robot efficient without losing accuracy:

1. Finding the "Spotlight" Layers (Focal-Guided Pruning)

Think of the AI model as a team of 30 or 40 detectives working together to solve a mystery (reading the text).

The Insight: The researchers found that not all detectives are equally good at looking at pictures. Some detectives (layers) are great at looking at the text, while a few specific detectives are the "experts" at spotting the visual details.
The Trick: FastOCR identifies these "Expert Detectives" (called Focal Layers).
- The Expert Detectives look at the entire page to find the most important words right now.
- The Regular Detectives (the rest of the team) don't need to look at the whole page. They just trust the Experts and only look at the small, important words the Experts pointed out.
Result: The team saves a massive amount of energy because most of them aren't staring at the whole firehose; they are just looking at the specific word the Experts highlighted.

2. The "Step-by-Step" Memory (Cross-Step Fixation Reuse)

Imagine you are reading a sentence: "The quick brown fox..."

When you read "The," your eyes are on the first word.
When you read "quick," your eyes move just a tiny bit to the right.
You don't need to re-scan the whole page to find "quick"; you just shift your gaze slightly from where you were a second ago.

FastOCR uses this logic:

When the robot finishes reading one word, it saves a note of exactly where it was looking.
For the very next word, it starts by looking at that same spot (or very close to it) before checking the rest.
Because the robot's eyes move smoothly from word to word, this "warm start" is almost always correct. It saves the robot from having to do a full search every single time.

The Results: Fast and Accurate

The paper tested this on several different AI models and found that:

Speed: The robot became 3 times faster at reading documents.
Accuracy: It kept 98% of its original accuracy. It didn't miss any words or get confused, even though it was only looking at 5% of the image data at any single moment.
Safety: Unlike the old methods that threw data away forever, FastOCR keeps the full image in memory. It just chooses to ignore most of it for the split second it's typing a letter. If it needs to look back, the data is still there.

The Bottom Line

FastOCR is like teaching a robot to read like a human: Focus on one word at a time, trust your memory of where you just were, and don't waste energy staring at the whole page when you only need to see a tiny spot. This makes reading documents incredibly fast without sacrificing the ability to get the details right.

Technical Summary: FastOCR

Problem Statement

Vision-Language Models (VLMs) have demonstrated strong capabilities in Optical Character Recognition (OCR), yet they face prohibitive inference costs when processing dense document images. Unlike general visual understanding tasks where a few salient patches suffice, OCR requires faithful transcription of every character, necessitating a massive number of visual tokens. This density leads to quadratic attention costs that scale poorly with sequence length.

Existing KV cache pruning methods rely on physical eviction, permanently discarding visual tokens during the prefill or encoding stages. While effective for natural images with semantic redundancy, this strategy is fundamentally flawed for OCR. Document images are information-dense; discarding even a small patch can irreversibly destroy character-level information, leading to catastrophic accuracy degradation (e.g., misreading "Classic Boat Museum" as "Glassed Bottle Museum"). Current methods fail to distinguish between the static nature of image content and the dynamic, sequential nature of human reading.

Methodology: FastOCR

The authors propose FastOCR, a training-free framework that accelerates VLM-based document parsing by recasting global pruning as a local, dynamic problem. The core insight is the observation of Dynamic Visual Fixation: although document images appear globally dense, the model's attention is temporally sparse. At each decoding step, the model concentrates on a small region that shifts gradually, mimicking how a human reader fixates on successive words rather than perceiving the entire page at once.

FastOCR avoids permanent information loss by retaining the full KV cache but dynamically selecting which tokens to attend to at each step. It comprises two complementary modules:

1. Focal-Guided Pruning (FGP)

FGP determines where the model looks at each decoding step. It operates on the observation that image attention is highly non-uniform across transformer layers.

Focal Layer Selection: During a lightweight warmup phase ( $w$ steps), the system profiles the image-attention ratio for each layer. It identifies a small, fixed set of focal layers ( $C$ ) that exhibit high concentration on image tokens. Notably, unlike general image understanding where attention is shallow, OCR focal layers reside in the middle and late stages of the network.
Cross-Layer Propagation: At every subsequent decoding step, only the focal layers attend over the full KV cache to select the top-attended image tokens (based on a kept ratio $\tau$ ). This selection is then propagated to all non-focal layers, which attend only to the pruned set of tokens. This avoids redundant full-sequence attention in layers that contribute little to visual understanding.

2. Cross-Step Fixation Reuse (CSFR)

CSFR guides how the model's fixation moves across time.

Mechanism: Since the attended region shifts only gradually between consecutive decoding steps, the focal tokens selected in the previous step serve as a strong prior for the current step.
Implementation: At the start of a new step, if the first layer is non-focal (which is common), it inherits the focal token set from the deepest focal layer of the previous step ( $F_{last}$ ) as a "warm start." This allows the model to initialize its attention efficiently without scanning the full cache immediately, refining the selection as soon as the first focal layer is reached.

Key Contributions

Identification of Dynamic Visual Fixation: The paper establishes that VLM-based OCR exhibits a phenomenon where document images are globally dense but temporally sparse in terms of attention. This reframes the pruning challenge from a static global selection to a dynamic local one.
FastOCR Framework: The proposal of a training-free, plug-and-play module combining Focal-Guided Pruning and Cross-Step Fixation Reuse. It dynamically attends to a small subset of visual tokens while keeping the full KV cache intact for future re-selection, avoiding the irreversible information loss of physical eviction.
Generalization and Efficiency: Extensive experiments demonstrate that FastOCR generalizes across five VLMs of varying sizes and architectures (including Qwen2.5-VL, dots.ocr, DeepSeek-OCR, olmOCR, and LLaVA-OneVision).

Experimental Results

The authors evaluated FastOCR on OmniDocBench and olmOCR-Bench, comparing it against physical-eviction baselines (FastV, VisionZip, H2O, PyramidKV).

Accuracy Retention: On Qwen2.5-VL, FastOCR retains 98% of the unpruned model's accuracy while attending to only 5% of visual tokens per decoding step. In contrast, physical eviction baselines suffer catastrophic degradation, often dropping to near-zero performance on table recognition (TEDS) and reading order tasks.
Latency Reduction: FastOCR achieves a 3.0× reduction in attention latency and a 1.4× end-to-end decoding speedup on Qwen2.5-VL. The attention latency speedup exceeds the theoretical FLOPs reduction because pruning reduces memory access costs, which dominate in memory-bound decoding regimes.
Robustness: The method remains effective across different models and document types (text, formulas, tables), whereas baselines like FastV and H2O fail to maintain utility on dense document tasks.

Significance and Claims

The paper claims that FastOCR offers a fundamental shift in how KV cache pruning is approached for OCR. By moving away from physical eviction (permanent discarding) toward dynamic selection (temporary pruning with full cache retention), it resolves the trade-off between efficiency and accuracy inherent in dense document parsing.

The authors emphasize that FastOCR is a training-free solution that can be deployed as a plug-and-play acceleration module. They acknowledge limitations, noting that peak memory consumption remains comparable to the unpruned model (as the full cache is retained) and that the current analysis is specific to dense document parsing, though the underlying "Dynamic Visual Fixation" phenomenon may extend to other information-dense vision-language tasks. The work suggests that future directions could include adaptive layer selection and application to other tasks like scene text recognition or chart understanding.