🤖 AI

On Locality and Length Generalization in Visual Reasoning

This paper demonstrates that while standard global vision models often fail to generalize across task complexity by exploiting shortcuts, recurrent policies based on strictly local perception can overcome these limitations and achieve robust compositional generalization, suggesting that local attention is essential for visual reasoning.

Original authors: Pulkit Madan, Sanjay Haresh, Reza Ebrahimi, Sunny Panchal, Apratim Bhattacharyya, Roland Memisevic

Published 2026-07-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Pulkit Madan, Sanjay Haresh, Reza Ebrahimi, Sunny Panchal, Apratim Bhattacharyya, Roland Memisevic

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to solve a giant, messy puzzle, but instead of looking at the whole picture at once, you have to use a tiny, high-powered magnifying glass to look at one piece at a time, moving your hand step-by-step across the table. That's exactly how our eyes work. We don't see the world in one giant, frozen snapshot; we take a series of quick, local "glimpses" to build a picture in our minds.

But here's the twist: most of the super-smart computer vision models we use today are like someone trying to solve that same puzzle by staring at the entire table from a distance, all at once. They take the whole image in one big gulp.

A new paper from researchers at Qualcomm AI Research asks a big question: Is that "one big gulp" approach actually holding computers back when it comes to really hard reasoning tasks? They set up a series of digital playgrounds to test this, and the results are pretty wild.

The "Parity" Puzzle and the "Shortcut" Trap

The researchers created a game called Visual Parity. Imagine a board with a bunch of light switches scattered around. Some are on (1), some are off (0). The goal? Figure out if the total number of "on" switches is an odd or even number.

For a human, this is easy: you just walk your eyes over the switches, count them up, and shout "Odd!" or "Even!" But for a computer model that sees the whole image at once, it's a trap. The paper shows that these "global" models (like the popular Qwen2.5-VL) are too clever for their own good. They learn a shortcut. Instead of actually counting every single switch, they might just look at the density of switches or the overall pattern of the image. It works perfectly when the board has, say, 5 to 10 switches (which is what they were trained on).

But the moment you throw a curveball—giving them a board with 15 or 20 switches, or making the board much bigger—the shortcut fails. The model panics. It's like a student who memorized the answers to a 10-question quiz but fails the 20-question test because they never actually learned how to count.

The paper explicitly rules out the idea that these models just need to be bigger or smarter. Even the most advanced models, when they try to see the whole image at once, hit a wall when the task gets longer or more complex. They simply can't generalize to the new, harder versions of the game.

The Secret Sauce: A "Foveated" Agent

So, what's the fix? The researchers built a robot agent called FOVEAGENT-LSTM. Instead of staring at the whole board, this agent is equipped with a "foveated" vision system, just like a human eye.

Think of it like this: The agent has a tiny, super-sharp camera in the center of its vision (the fovea) that sees details clearly, and a blurry, wide-angle camera around the edges (the periphery) that helps it know where to move next. It takes a step, looks at one switch, updates its mental state, moves to the next, and repeats.

This "step-by-step" approach is the key. The paper finds that when the agent is forced to look at the image locally, one piece at a time, it actually learns the logic of the task. It doesn't rely on shortcuts. When the researchers tested this agent on boards with way more switches than it ever saw during training (up to 20 switches!), it kept getting it right. It suggests that local perception (looking at small pieces) combined with recurrence (remembering what you saw in the previous step) is the magic recipe for solving these problems.

What About Just "Finding" Things?

Here's where it gets interesting. The researchers also tested a different kind of game called Recall. Imagine a room full of red "T"s and green "L"s, and you have to find a red "L". This is a search task, not a counting task.

In this scenario, the "global" models (the ones that see everything at once) actually did great! They didn't need to walk around the room step-by-step. They could just scan the whole room and spot the target. The paper suggests that for simple "search and find" tasks, you don't need the complex, step-by-step local vision. You only need that step-by-step approach when you have to track a state (like counting or following a sequence of rules).

The Real-World Test: Finding Roots

To see if this works in the real world, they tried a math task: finding the "roots" (where a line crosses zero) on a graph. They had to find these roots in plots that were cluttered with other lines and subplots.

When they used a standard global model, it struggled to find the roots in the messy, complex graphs. But when they used their "foveated" agent (which looked at the graph in small, high-resolution chunks), it performed much better. The paper suggests that for tasks requiring you to piece together information from different parts of an image, looking locally is a huge advantage.

The Catch: Size Matters

There's a delicate balance, though. The researchers found that if the "glimpse" the agent takes is too big, it starts to cheat again. If the agent can see too many switches at once, it stops counting and starts guessing patterns again.

They found a "sweet spot." The agent needs a high-resolution view of the specific switch it's looking at (to see if it's on or off) but a low-resolution, blurry view of the surrounding area (to navigate without seeing too much). If the blurry view is too sharp, the agent learns shortcuts. If it's too blurry, the agent gets lost. It's a Goldilocks situation: the peripheral view needs to be just right to help the agent move without letting it cheat.

The Bottom Line

The paper doesn't claim to have solved all of AI's problems. It suggests that for tasks where you need to count, track, or follow a sequence of steps across a large image, the current "look-at-everything-at-once" models might be hitting a ceiling.

It suggests that to get truly robust, human-like reasoning in computers, we might need to go back to basics and build models that "look" at the world the way we do: one focused glimpse at a time, remembering what came before, and moving step-by-step through the puzzle. It's a reminder that sometimes, seeing less (locally) actually helps you understand more (globally).

Technical Summary: On Locality and Length Generalization in Visual Reasoning

Problem Statement

Current state-of-the-art vision models, particularly those based on Transformers, typically ingest images globally in a single forward pass. While these models achieve human-level performance on tasks like image captioning and visual question answering, they rely on global computations that differ fundamentally from human vision, which processes information through a series of local, foveated glimpses connected by saccades.

The central problem addressed in this work is length generalization in visual reasoning: the ability of a model to solve tasks requiring the aggregation of local information across an image when the number of entities or the complexity of the task exceeds the training distribution. Drawing parallels to findings in language models (e.g., Dubois et al., 2020; Anil et al., 2022), the authors investigate whether global vision models fail to generalize to longer, more complex visual sequences because they learn "global shortcuts" rather than true compositional reasoning strategies. The paper posits that unlike textual tasks where information is already decomposed into sequential tokens, visual tasks require the model to actively gather and aggregate spatially distributed information, a process potentially hindered by global perception.

Methodology

Test-Bed for Length Generalization

The authors introduce a suite of visual reasoning tasks designed to evaluate extrapolation capabilities:

Visual Parity: A visual analog to the binary parity problem. The model must locate $N$ binary switches (0 or 1) distributed randomly on a 2D canvas and determine if the sum of pressed states is odd or even. This tests permutation-invariant aggregation.
State Machine: A task requiring order-sensitive state updates. Switches trigger rules (increment/decrement) on a numerical state, with arrows indicating the traversal order. This tests sequential reasoning and state tracking.
Recall: A control task involving visual retrieval (finding a specific object among distractors) without state tracking, designed to test if generalization failures are specific to state tracking or apply to all visual tasks.
Finding Roots: A real-world mathematical reasoning task requiring the localization of specific subplots and functions to identify roots, testing performance on complex, structured data.

The FOVEAGENT-LSTM Model

To test the hypothesis that local perception and recurrence are necessary for generalization, the authors propose FOVEAGENT-LSTM, a recurrent vision model inspired by biological foveation.

Input Mechanism: At each time step, the model receives two inputs: a high-resolution foveated glimpse ( $G_l$ ) for fine-grained detail and a low-resolution peripheral glimpse ( $G_p$ ) for navigation.
Architecture: The glimpses are encoded by ResNet encoders and concatenated, then passed to an LSTM backbone.
Action Space: The model outputs a displacement vector (angle and magnitude) to determine the next gaze location and a stop bit to signal task completion.
Probes: Intermediate supervision is provided via "probes" (e.g., the state of the current switch) to mimic chain-of-thought reasoning and improve data efficiency.

Experimental Setup

The authors compare FOVEAGENT-LSTM against strong baselines, including QWEN2.5-VL-3B-INSTRUCT (fine-tuned) and various closed-source models (GPT-4o, Claude Sonnet). Experiments are conducted on In-Distribution (InD) data (e.g., 2–10 switches) and Out-of-Distribution (OOD) data (e.g., 11–20 switches, varying resolutions). The study systematically ablates:

Perception Type: Global view vs. Local+Global vs. Strictly Local (Foveated).
Architecture: Recurrent (LSTM) vs. Non-recurrent (Transformers, Linear RNNs like Mamba/XLSTM).
Glimpse Parameters: The size and resolution of the peripheral and foveated views.

Key Contributions

Length Generalization as a Visual Benchmark: The paper establishes visual reasoning tasks requiring the aggregation of local information as a fundamental testbed for length generalization, highlighting a gap in current evaluations that focus on in-distribution performance.
Identification of Global Shortcuts: The authors demonstrate that current vision and vision-language models, despite strong in-distribution performance, fail to extrapolate to longer visual sequences. This failure is attributed to the models learning global shortcuts that break under increased visual complexity.
The Necessity of Locality and Recurrence: The work traces the failure to two specific factors:
- Global Perception: Models with access to the full image (even recurrent ones) fail to generalize OOD, suggesting that seeing the whole image allows for shortcut learning.
- Non-Recurrence: Models with local perception but non-recurrent backbones (Transformers, Linear RNNs) also fail to generalize.
- Conclusion: Both strictly local perception and recurrent processing are necessary and jointly sufficient for robust OOD generalization in state-tracking tasks.
Distinction Between Recall and State Tracking: The paper shows that while global models fail at state tracking, they succeed at "recall" tasks (visual retrieval without state tracking), confirming that the generalization failure is specific to sequential state aggregation.
Real-World Validation: The findings are validated on a real-world mathematical reasoning task (Finding Roots), showing that foveated approaches significantly outperform global models in OOD settings, even when compute budgets are matched.

Results

Generalization Performance: FOVEAGENT-LSTM maintains high accuracy on OOD tasks (e.g., 11–20 switches) where global models (QWEN2.5-VL) show a sharp decline in performance.
Ablation of Perception: Models trained with access to the full high-resolution image (Global or Local+Global setups) fail to generalize OOD, whereas the strictly foveated model succeeds. This supports the claim that global perception enables shortcut learning.
Ablation of Architecture: When fed the same sequence of local glimpses, non-recurrent architectures (Transformers, Mamba, XLSTM) fail to generalize, while the LSTM succeeds. This confirms that recurrence is essential for the step-by-step computation required for state tracking.
Glimpse Size Trade-off: There is an optimal balance for glimpse size and resolution. Larger glimpses allow for faster processing but encourage shortcut learning; smaller glimpses force sequential processing but may hinder navigation. A high-resolution fovea combined with a low-resolution periphery resolves this conflict.
Active Search: The model successfully generalizes even when explicit traversal cues (arrows) are removed, provided it has mechanisms for active search and spatial memory (marking visited switches).
Compute Efficiency: In the "Finding Roots" task, FOVEAGENT-QWEN (a foveated version of Qwen) nearly doubles the accuracy of the global baseline for the same visual compute budget, demonstrating that allocating compute to high-resolution local regions is more effective than uniformly increasing global resolution.

Significance and Claims

The paper argues that local attention is an essential, yet overlooked, requirement for robust compositional generalization in visual reasoning. The authors claim that the standard practice of encoding visual information in a global context window (via ViTs or CNNs) prevents models from developing the inductive biases necessary for extrapolation.

The significance of this work lies in:

Biological Plausibility: It suggests that the sequential, foveated nature of human vision is not merely an evolutionary artifact but a computational necessity for handling variable-length, complex reasoning tasks.
Architectural Guidance: It proposes that future visual foundation models aiming for human-level reasoning capabilities may need to return to biologically inspired, sequential attention policies that combine recurrence with strictly local perception.
Limitations of Current Benchmarks: It highlights that standard benchmarks, which often rely on in-distribution performance, fail to capture the inability of current models to generalize to unseen levels of complexity or length.

The authors remain modest in their claims, noting that while they demonstrate the necessity of locality and recurrence for state tracking, these mechanisms are not required for recall tasks. Furthermore, they acknowledge that learning general policies for local perception in real-world scenarios (beyond synthetic tasks) remains an open research problem requiring reinforcement learning or more diverse training data.