💻 computer science

Q-Mask: Query-driven Causal Masks for Text Anchoring in OCR-Oriented Vision-Language Models

This paper introduces Q-Mask, a novel OCR framework that employs a causal query-driven mask decoder to explicitly ground text to spatial regions before recognition, supported by the TextAnchor-Bench benchmark and the large-scale TextAnchor-26M dataset to significantly improve text anchoring capabilities in vision-language models.

Original authors: Longwei Xu, Feng Feng, Shaojie Zhang, Xin Chen, Hang Li, Anan Du, Hailong Yu, Pei Fu, Zhenbo Luo, Jian Luan

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Longwei Xu, Feng Feng, Shaojie Zhang, Xin Chen, Hang Li, Anan Du, Hailong Yu, Pei Fu, Zhenbo Luo, Jian Luan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific word in a crowded, messy room full of people shouting different things.

The Problem:
Current "smart" AI models (Vision-Language Models) are like brilliant librarians who can read any book perfectly. If you show them a picture of a receipt, they can tell you the total price or the date. However, they have a weird blind spot: they can't point to where the text is.

If you ask, "Where is the word 'Total'?", a standard AI might say, "It's 12.50," but it can't draw a box around it. It's like a librarian who knows the answer but refuses to point to the shelf where the book sits. This is a problem for things like smart glasses or robots that need to interact with the physical world—they need to know where to look, not just what to read.

The Solution: Q-Mask
The researchers at Xiaomi created a new system called Q-Mask. Think of it as teaching the AI a new way of thinking: "Look before you read."

Instead of trying to read the whole image and guess the answer at the same time, Q-Mask breaks the process into two distinct steps, similar to how a detective solves a case:

The "Where" Step (The Search): First, the AI looks at your question (e.g., "Where is the price?") and scans the image to draw a mental "spotlight" or a mask over the specific area where that information likely lives. It ignores everything else.
The "What" Step (The Reading): Only after it has isolated that specific spot does it zoom in and read the text inside that spotlight.

The Creative Analogy: The "Flashlight" vs. The "Floodlight"

Old AI (Floodlight): Shines a bright light over the entire messy room. It sees everything at once, gets overwhelmed, and sometimes confuses a word on a poster with a word on a menu. It guesses the answer based on the whole scene.
Q-Mask (Flashlight): You ask, "Where is the exit sign?" The AI first uses a flashlight to find the exit sign. Once the flashlight is locked onto the sign, then it reads the letters. This ensures the answer is grounded in reality.

How They Taught It (The Training Data)
To teach the AI this new skill, the researchers couldn't just use normal books. They needed a special training manual called TextAnchor-26M.

Imagine trying to teach a child to find hidden objects. You wouldn't just show them a picture and ask "What do you see?" You would show them a picture, point to a specific spot, and say, "The word 'Apple' is right here."

They created 26 million examples of this.
They used a trick called "De-stylized Mask Rendering." Instead of showing the AI the exact, fancy font from a real sign (which might be hard to read), they took the text and printed it in a plain, boring font on a white background, matching the shape of the original sign. This taught the AI to focus on the shape and location of the text, not the fancy design.

The Result: The "TextAnchor-Bench" (TABench)
The team also built a test called TABench to see if other AIs could do this. They found that even the most famous, powerful AI models (like GPT-4o or Gemini) were terrible at pointing to text. They could read it, but they couldn't anchor it to a location.

Q-Mask, however, passed the test with flying colors. It became much better at:

Reading: Getting the text right because it focused on the right spot.
Pointing: Accurately drawing a box around the text.

Why This Matters
This isn't just about reading better. It's about interaction.

Smart Glasses: If you wear glasses that tell you what a sign says, you want the glasses to highlight the sign for you, not just speak the words.
Robots: If a robot needs to pick up a box labeled "Fragile," it needs to know exactly where that label is on the box to grab it safely.

In Summary:
Q-Mask is a new way of teaching AI to stop guessing and start searching. By forcing the AI to find the location of the text before reading it, the researchers created a system that is more accurate, more reliable, and much better at understanding the physical world. It's the difference between a student who guesses the answer on a test and a student who shows their work by pointing exactly to where the answer is found.

1. Problem Statement

While modern Vision-Language Models (VLMs) have advanced significantly in Optical Character Recognition (OCR) and Visual Question Answering (VQA), they often lack reliable text anchoring.

The Gap: Current models can often read text or locate regions, but they struggle to explicitly and stably ground a specific queried text token to its corresponding spatial region in an image.
The Limitation of Existing Methods:
- Standard VLMs: Trained end-to-end on image-text pairs, they learn to recognize text but remain unconcerned with the precise spatial location (anchor) of that text.
- Existing Mask-Based Methods: Some approaches use character masks as supervision, but they often condition mask prediction on the entire question-answer sequence (including the answer tokens). This creates a non-causal dependency where the spatial prior is not available before the model generates the answer, failing to enforce a "look before read" reasoning process during inference.

2. Methodology: Q-Mask Framework

The authors propose Q-Mask, a framework designed to explicitly construct text anchors through a structured, causal reasoning process.

A. Causal Query-Driven Mask Decoder (CQMD)

The core innovation is the CQMD, which enforces a Visual Chain-of-Thought (CoT) paradigm.

Causal Decoding: Unlike standard models that generate masks based on the full answer sequence, CQMD predicts spatial masks solely from the image tokens and query tokens. It explicitly excludes answer tokens from the mask generation process.
Mechanism:
1. The model extracts hidden states for visual tokens ( $H_{img}$ ) and query tokens ( $H_q$ ).
2. Using cross-attention, it computes query-aware spatial features ( $S$ ) that represent the likely location of the text mentioned in the query.
3. These features are upsampled to generate a segmentation mask ( $\tilde{M}$ ) before the autoregressive generation of the final text answer.
Mathematical Formulation: The model optimizes for $P(A|I, Q) = \int P(A|S, Q)P(S|I, Q)dS$ , where $S$ is a latent spatial support variable independent of future answer tokens ( $S \perp A_{future}$ ).

B. Training Objective

The model is trained with a joint loss function:
$L_{SSA} = \lambda_{txt} L_{NTP} + \lambda_{seg} L_{mask}$

$L_{NTP}$ : Standard Next-Token Prediction loss for text generation.
$L_{mask}$ : A combination of Dice loss and Cross-Entropy loss to supervise the predicted segmentation mask against ground-truth masks.

C. TextAnchor-26M Dataset

To train the CQMD, the authors constructed TextAnchor-26M, a large-scale dataset (~26.7M instances) containing fine-grained image-text pairs annotated with bounding boxes and masks.

Sources: Unconstrained scene text (mined from web corpora), academic documents (arXiv), synthetic text (rendered with diverse fonts), and VQA pairs.
De-stylized Mask Rendering: Instead of relying on pixel-perfect glyph contours, the dataset uses "de-stylized" masks. Text is rendered on a blank canvas using standard fonts and aligned to the bounding box. This reduces sensitivity to low-level textures and encourages modality-agnostic alignment between linguistic content and spatial support.
Stochastic Prior Injection (SPI): To simulate real-world OCR variability, the training data injects noisy priors (missing text, jittered boxes, or corrupted transcripts) based on empirical error profiles, forcing the model to be robust to imperfect spatial cues.

3. Benchmark: TextAnchor-Bench (TABench)

To systematically evaluate text anchoring, the authors introduced TABench.

Tasks:
1. Region-to-Text (R2T): Given a region, read the text.
2. Text-to-Region (T2R): Given a text query, locate the region.
Composition: 5,450 queries across 970+ images, covering 12 scenarios (e.g., receipts, scene text, documents) with balanced R2T and T2R tasks.
Metrics: Exact-match accuracy for R2T and F1-score (IoU > 0.5) for T2R.

4. Key Contributions

Q-Mask Framework: A novel architecture that decouples "where the text is" from "what the text is" by enforcing a causal, query-driven mask generation step prior to answer decoding.
TextAnchor-26M: A massive, high-fidelity dataset providing fine-grained spatial supervision for training VLMs to establish stable text anchors.
TABench: A comprehensive benchmark revealing that even state-of-the-art VLMs (including GPT-4o, Gemini, and Qwen) struggle with bidirectional text anchoring, highlighting a critical gap in current models.
Stochastic Prior Injection (SPI): A data augmentation strategy that simulates realistic OCR errors to improve model robustness.

5. Experimental Results

Performance on TABench: Q-Mask (3B parameters) achieved an Overall score of 45.50%, significantly outperforming the Qwen3-VL-2B baseline (37.77%) and rivaling much larger models. It demonstrated a balanced improvement in both R2T and T2R tasks.
Downstream VQA: On standard benchmarks like TextVQA, Q-Mask-3B achieved 88.5, a 9.2-point improvement over its baseline. It outperformed specialized OCR models (e.g., TokenVL-8B) and rivalled generalist models twice its size (e.g., InternVL3.5-8B).
Ablation Studies:
- Removing CQMD resulted in significant drops in performance, confirming the necessity of the causal mask decoder.
- SPI improved the balance between localization and recognition, reducing false negatives in T2R tasks.
- Scaling the Stage-1 spatial supervision showed a strong correlation with downstream performance, with optimal stability observed above 4M samples.

6. Significance

The paper addresses a fundamental limitation in current VLMs: the lack of explicit, reliable spatial grounding for text. By introducing a causal, query-driven reasoning process, Q-Mask forces the model to "look" at the relevant region before "reading" the text. This approach:

Improves Interpretability: The generated masks serve as explicit intermediate representations of the model's reasoning.
Enhances Robustness: The model becomes more reliable in complex scenes with dense text or cluttered backgrounds.
Sets a New Standard: TABench provides a rigorous evaluation protocol for future research into grounded reasoning, moving beyond simple text recognition to precise text-region correspondence.