💬 NLP

Learning to Look Again: Loss-Gap Supervision for Free-form Crop Routing in Vision-Language Models

The paper proposes GapSight, a framework that trains a lightweight router to enable Vision-Language Models to selectively re-examine free-form image regions based on their own failure signals (loss-gap supervision), thereby significantly improving performance on detail-centric tasks without indiscriminately increasing computational costs.

Original authors: Jinchang Zhu, Rong Fu, Yi Ding, Chenghao Wu, Ying Liu, Menglin Yang

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jinchang Zhu, Rong Fu, Yi Ding, Chenghao Wu, Ying Liu, Menglin Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern computers have become remarkably good at looking at pictures and answering questions about them. These systems, known as vision-language models, act as general assistants that can read a document, describe a chart, or identify an object in a photograph. They work by taking an image, compressing it into a digital summary, and then using that summary to generate an answer. However, a persistent problem remains: when these models compress a picture to make it easier to process, they often lose the tiny, crucial details needed to answer specific questions. A receipt might become a blur of gray, a chart's numbers might collapse into indistinguishable smudges, or a small label on a map might vanish entirely. The answer is often right there in the original image, but it is inaccessible once the picture has been simplified.

For years, engineers tried to solve this by simply feeding the computer more visual information. They would break images into many small pieces or show the model the same picture at high resolution repeatedly. While this helped with tasks like reading text, it was a blunt instrument. It forced the computer to spend extra time and energy on every single question, even when the simple, low-resolution view was already enough to find the answer. It was like using a high-powered microscope to read a street sign; the detail was there, but the effort was wasted on questions that didn't need it. The challenge was to teach the model to know when to look closer and when to trust its initial glance.

A team of researchers has developed a new approach called GapSight that teaches these models to learn when and where to look again. Instead of forcing the computer to examine every image in high detail, GapSight trains the model to take a quick, global look first. If the model senses that the question requires evidence it cannot see in that initial view, it learns to pause and select a specific, free-form region of the image to examine more closely. This decision is not made by a human programmer or a separate rulebook; the model learns this behavior by observing its own mistakes.

The training process works by simulating a "second look" during the learning phase. The researchers take a question and an image and ask the model to answer it using only the low-resolution view. They then generate many different candidate crops, which are small, zoomed-in sections of the image, and ask the model to answer the same question again using these zoomed-in views. If a specific zoomed-in section helps the model give a better answer or feel more confident in its choice, that section is marked as useful. If a zoomed-in section does not improve the answer, it is marked as unnecessary. Over time, the model learns to recognize the difference between a question that needs a closer look and one that does not, based entirely on whether the extra visual detail actually changes the outcome.

Once trained, this system operates with a lightweight decision-maker attached to the main model. When a new image arrives, the model takes its first global glance. The decision-maker then predicts whether the model should stick with that initial view or inject a single, carefully chosen crop for a second look. This crop is not restricted to a fixed grid or a standard shape; it can be any size or shape, allowing it to wrap around a block of text, a specific chart area, or a small object. The system decides this before it even sees the zoomed-in pixels, relying only on the context of the global image and the question being asked.

The results of this method show a clear advantage over previous techniques. When tested on six different benchmarks covering tasks like reading receipts, interpreting charts, and analyzing infographics, the new system significantly outperformed models that simply looked at the whole image or those that used fixed, always-on zooming methods. On one specific model, the average score across these six tasks jumped from 52.25 to 64.29. This improvement was achieved while using fewer visual resources than the older methods, proving that the system learned to allocate its attention efficiently. It did not just add more data; it learned to add the right data at the right time.

The researchers also discovered that the best place to look is often specific to the model itself. A region that helps one computer model answer a question correctly might not help another, because different models process visual information in slightly different ways. This finding ruled out the idea of a universal "best crop" that works for all systems. Instead, the system must learn its own strategy for re-reading based on its own internal strengths and weaknesses. The study showed that this learned behavior is highly adaptable: the model reviews images frequently when the task involves reading dense text or complex infographics, but it largely abstains from looking closer when the question relies on the overall layout or global context of a scene.

This approach represents a shift from brute-force processing to intelligent allocation. By teaching the model to measure the value of a second look before taking it, the system avoids wasting energy on questions that are already solved. It rescues concrete errors where local details were missing, such as a specific number on a sign or a value on a graph, without disturbing the global understanding of the image. The work demonstrates that the key to solving detail-oriented problems in artificial intelligence is not necessarily to see more, but to know exactly when and where to look again.

Technical Summary: Learning to Look Again via Loss-Gap Supervision

Problem Statement

Vision-Language Models (VLMs) frequently fail on detail-centric questions because the visual evidence required to answer is present in the image but lost during the compression of the image into a low-resolution global view. While allocating more visual tokens (e.g., via high-resolution tiling or constant crop augmentation) can improve performance on OCR-heavy tasks, it indiscriminately increases computational costs and can disrupt tasks that rely on global spatial context.

The core challenge is supervision: determining when a model needs to re-read an image and where to look. Useful crops are model-specific; a region that aids one VLM may not help another due to differences in encoders, tokenizers, and decoders. Existing training-free methods often rely on external signals (like CLIP relevance) that do not directly reflect answer quality, while human annotations are costly and may not align with the specific evidence a model needs (e.g., layout context vs. object bounding boxes).

Methodology: GapSight

The authors propose GapSight, a framework that learns a visual re-reading policy through loss-gap supervision. The system operates in two phases: offline label mining and online one-shot inference.

1. Loss-Gap Supervision (Offline Mining)

Instead of using external annotations, GapSight mines supervision signals directly from the target VLM's own behavior.

Process: For a given image-question pair, the system compares the model's answer behavior under a global-only view versus candidate crop-augmented views.
Signal Generation:
- For Generated Answers: The utility of a crop is measured by the reduction in the negative log-likelihood (NLL) of the target answer span.
- For Multiple-Choice Benchmarks: The utility is measured by the increase in the margin between the correct option's score and the highest incorrect option's score.
Label Creation: A candidate crop that produces a significant positive "loss gap" (improvement) is selected as the optimal crop ( $b^*$ $b^{*}$ ) for that specific example and model. This creates a training label consisting of:
- Action: review (if the gap is positive) or preserve (if not).
- Utility: The magnitude of the improvement ( $\Delta^*$ ).
- Box: The coordinates of the best crop ( $b^*$ ).

2. Free-Form Crop Router

A lightweight router is trained to predict the re-reading policy based solely on the global image state (before any crop tokens are rendered).

Inputs: Hidden states from the global preview.
Outputs:
1. Gate Logit: A binary decision to preserve the global view or inject a crop.
2. Utility Score: A scalar estimate of the expected benefit.
3. Continuous Box: A normalized bounding box $(c_x, c_y, w, h)$ that is not confined to a fixed grid, allowing it to adapt to text blocks, chart areas, or objects.
Training: The router is trained on dual-path examples. "Preserve" rows supervise the gate to abstain, while "Review" rows supervise the gate, utility, and box heads using the mined labels. The VLM backbone remains frozen; only the router and its auxiliary heads are trained.

3. Inference

At inference time, the model processes the global preview. The router predicts the action. If review is selected, a single crop is rendered and injected into the context (global + crop), and the final answer is generated. This constitutes a one-shot re-reading process.

Key Contributions

Loss-Gap Supervision: The introduction of a supervision signal derived from the target model's own answer behavior (NLL reduction or option-margin improvement) to determine useful visual evidence. This aligns supervision with the specific capabilities and failures of the target backbone.
One-Shot Free-Form Router: A lightweight module that predicts whether to re-read, the expected utility, and a continuous crop box from the global state alone, enabling dynamic visual budget allocation without iterative search.
Cross-Backbone Evaluation: Demonstration of the framework across three distinct VLM families (LLaVA-1.5-7B, InternVL2.5-8B, and Qwen2-VL-2B-Instruct) and six diverse benchmarks, showing that crop utility is model-specific and requires tailored supervision.

Experimental Results

The authors evaluated GapSight on six benchmarks: TextVQA, DocVQA, ChartQA, InfographicVQA, VStarBench, and MME-RealWorld-Lite.

Performance Gains: GapSight consistently outperforms the "Base no-zoom" baseline and external crop/zoom methods (ZoomRefine, ViCrop, CropVLM).
- On InternVL2.5-8B, the average score across six benchmarks increased from 52.25 (Base) to 64.29, surpassing the next best method (CropVLM at 57.16).
- On LLaVA-1.5-7B, the average improved by 9.81 points over the baseline.
- On Qwen2-VL-2B-Instruct, the average rose from 44.65 to 56.02.
Token Efficiency: GapSight achieves these gains with a lower average visual token count than methods that indiscriminately add crops. For InternVL2.5-8B, it uses 391.0 tokens on average compared to ~512 for other crop methods, while achieving higher accuracy.
Mechanism Analysis:
- Model Specificity: Transferring mined crops between different backbones resulted in significant utility drops (off-diagonal retention averaged 0.44), confirming that useful crops are specific to the target model's architecture.
- Error Correction: On InternVL2.5-8B, GapSight corrected 107 errors from the baseline while introducing only 27 regressions.
- Adaptive Behavior: The router learns to review frequently on text-heavy tasks (e.g., 76.5% action rate on InfographicVQA) but abstains on tasks requiring global context (e.g., 25.6% on MME-Lite).

Significance and Claims

The paper positions loss-gap supervision as a practical route to teaching VLMs "when and where to look again." The authors claim that:

Alignment: Supervision derived from the target model's own failure signals is superior to external heuristics (like CLIP relevance) or random sampling.
Separation of Concerns: The method cleanly separates the learning of visual re-reading from answer generation, allowing the VLM to retain its standard generative interface while the router manages visual evidence allocation.
Controllable Trade-off: The approach exposes a continuous cost-performance tradeoff, where practitioners can adjust gate and utility thresholds to control the visual budget.
Context Preservation: By predicting free-form, context-expanded boxes, the method ensures that local details (e.g., a number) are viewed alongside necessary context (e.g., its unit or axis), avoiding the pitfalls of isolated cropping.

The results suggest that visual re-reading is not merely about increasing resolution, but about learned allocation of visual tokens based on the specific evidence required to resolve a query.