💻 computer science

SEER: A Self-Grounded Evidence Interface for Controlled Spatial Relation Classification

The paper introduces SEER, a training-free inference-time interface for frozen Vision-Language Models that improves spatial relation classification by constructing query-specific evidence views with explicit subject/object roles and complementary geometric context, thereby significantly reducing errors caused by ambiguous global views or incorrect instance selection.

Original authors: Feixiang Liu, Likun Wang, Qiang Qiu, Hui Xu, Huawei Shen, Xueqi Cheng

Published 2026-08-05

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Feixiang Liu, Likun Wang, Qiang Qiu, Hui Xu, Huawei Shen, Xueqi Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a puzzle where two pieces need to fit together perfectly. In the world of artificial intelligence, there are "Vision-Language Models" (VLMs)—super-smart computer brains that can look at a picture and read a question about it. They are great at recognizing objects; if you show them a photo of a cat and a dog, they know which is which. But here is the tricky part: when you ask them, "Is the cat to the left of the dog?", they sometimes get confused. They might see the animals correctly but mix up their positions, or they might guess based on what they've heard in books rather than what they actually see. This is like a student who knows the definitions of "left" and "right" but keeps pointing the wrong way when looking at a map. Scientists care about fixing this because if these AI brains can't understand spatial relationships, they can't help us with things like driving cars, navigating robots, or reading complex diagrams safely.

Enter SEER, a clever new trick for these AI brains that doesn't require teaching them anything new. Think of SEER as a "spotlight and label" system. Instead of letting the AI stare at the whole messy picture and guess, SEER first quietly finds the two specific things the question is asking about (like the cat and the dog). Then, it creates a special, zoomed-in view just for those two items. It draws a red box around the subject and a blue box around the object, labeling them clearly as "Subject" and "Object." It hides the possible answers (like "left" or "right") during this step so the AI doesn't rely on the answer choices to decide where to look. By forcing the AI to focus on a clean, labeled view of just the two items, SEER helps the AI make a much better decision. The researchers found that this simple "refocus and label" method made the AI significantly better at getting the spatial relationships right, improving its accuracy by about 4% on average across different tests.

The Problem: The AI's "Giant Blur"

Imagine you are in a crowded room full of people, and someone asks, "Is the person in the red hat standing to the left of the person in the blue shirt?" If you just glance at the whole room, you might get it wrong. Maybe you see a red hat on the far left and a blue shirt on the far right, but the specific people the question is about are actually standing right next to each other in the middle.

Current AI models often make this same mistake. They can see the objects, but they get lost in the "global view" of the image. They might rely on what they think usually happens (like "cats are usually on the left") instead of actually checking the specific spots in the picture. This is called a "hallucination" or a "failure to ground" the answer in the right visual evidence. The paper argues that the problem isn't that the AI is too dumb to understand "left" or "right," but that it's looking at the wrong part of the picture or getting confused by too much background noise.

The Solution: SEER's "Self-Grounded" Flashlight

The authors of this paper, working at the Institute of Computing Technology in China, came up with a method called SEER (Self-grounded Evidence for Entity-Relation Reasoning). It's like giving the AI a flashlight and a pair of highlighters, but with a very strict rule: Don't look at the answer key yet.

Here is how SEER works, step-by-step:

The "Blind" Search: When the AI gets a question like "Is the mug to the left of the book?", SEER first asks the AI to find the mug and the book in the image. Crucially, it hides the possible answers (left, right, above, below) during this search. This prevents the AI from relying on the word "left" and then just finding a mug on the left side of the image, regardless of where the book actually is.
The "Zoom-In" View: Once the AI finds the mug and the book, SEER cuts out a small picture containing just those two items. It draws a red box around the mug (the subject) and a blue box around the book (the object). It labels them clearly.
The "Role-Explicit" Check: Now, the AI looks at this clean, zoomed-in picture. It sees: "Okay, the red box is the mug, the blue box is the book. Now, is the red box to the left of the blue box?" Because the background is gone and the roles are labeled, the AI is much less likely to get confused.
The "Double-Check" (Optional): Sometimes, the AI might still be unsure. SEER has a neat trick called "reciprocal consistency." It swaps the roles: "Okay, now pretend the book is the subject and the mug is the object. Is the book to the right of the mug?" If the AI's answer makes sense in both directions (if A is left of B, then B must be right of A), it confirms the answer. If the answers contradict, it knows something is wrong and tries again.

What the Experiments Showed

The researchers tested this method on several different datasets, which are like giant collections of picture-and-question pairs. They used a "frozen" test set, meaning they picked the test questions before they started running the experiments, so they couldn't accidentally tweak their method to just memorize the answers.

The Big Win: On a test set called GQA-Train900 (which has 900 unique images), SEER improved the AI's accuracy by +3.94% compared to just looking at the full image. That might sound small, but in the world of AI benchmarks, a nearly 4% jump is a huge deal. It means the AI got almost 40 more questions right out of 1,000.
Different Models, Same Success: They tried this on different types of AI brains (like Qwen3 and InternVL3.5). Every single one got better. For example, on a different test called EmbSpatial, one model improved by a massive +11.79%.
Why It Works: The researchers ran special tests to figure out why it worked. They found that the magic wasn't just "zooming in" (cropping the image). It was the labels. When they cropped the image but didn't label which was the subject and which was the object, the AI didn't improve as much. The clear "Red Box = Subject" and "Blue Box = Object" labels were the key. It forced the AI to pay attention to the specific roles of the objects.
What It Didn't Fix: The method isn't perfect. It struggled a bit with the word "on" (like "the cup is on the table") because "on" is hard to tell apart from just being "above" without 3D depth. It also didn't help as much when the AI had to guess "true" or "false" on binary questions, suggesting that the trick works best when there are specific choices to pick from.

The Takeaway

SEER proves that you don't always need to build a bigger, smarter, or more expensive AI to solve a problem. Sometimes, you just need to change how you show the problem to the AI. By hiding the answer choices during the search and then presenting a clean, labeled view of just the relevant objects, the AI can stop guessing and start seeing.

The authors are careful to say this isn't a magic wand that solves every spatial problem. The AI still needs to be good at finding the objects in the first place. But when the objects are found, SEER acts like a helpful guide, ensuring the AI looks at the right thing in the right way. It's a reminder that in the race for smarter AI, sometimes the best upgrade is a better interface.

Technical Summary: SEER – A Self-Grounded Evidence Interface for Controlled Spatial Relation Classification

Problem Statement

Vision-Language Models (VLMs) frequently fail at spatial relation reasoning, even when they can correctly recognize individual entities. Benchmarks such as VSR, What'sUp, GQA, and ARO demonstrate that models often recognize objects but fail to compare their positions, relying instead on language priors, attending to the wrong instances, or interpreting ambiguous global views. Current reasoning-oriented checkpoints and multimodal models do not inherently guarantee the selection of correct visual evidence. The core issue is that a spatial answer is only as reliable as the input evidence; if the model attends to an incorrect instance or lacks explicit role conditioning, the premise for relation prediction is corrupted before the decision is made.

Methodology: SEER

SEER (Self-grounded Evidence for Entity-Relation Reasoning) is a training-free, inference-time evidence interface designed for frozen VLMs. It does not train new models or rely on external detectors. Instead, it constructs query-specific visual evidence states to guide the VLM's decision process. The method operates in four stages:

1. Target-Relation-Hidden Self-Grounding

SEER first localizes the queried subject ( $S$ ) and object ( $O$ ) pair using a frozen VLM grounder ( $G$ ). Crucially, this grounding step is target-relation-hidden: the model receives the image and entity names but is explicitly withheld the candidate relations and answer options. This prevents the grounder from biasing box selection toward a specific relation.

Validation: Boxes are validated for positive area and valid coordinates. If grounding fails, the system falls back to the full image.

2. Construction of Complementary Evidence States

SEER generates three distinct evidence types for the same subject/object pair:

Full Image Evidence: The original image and question, preserving global context. This serves as the safe fallback.
Self-Grounded Marker View: A cropped view containing the union of the subject and object bounding boxes. The subject is marked with a red box labeled "S," and the object with a blue box labeled "O." This state provides local refocus and explicit role assignment, forcing the model to attend to the specific pair and their linguistic roles without candidate relations.
Threshold-Eligible Geometry: A deterministic candidate derived from the box centers and extents. It is eligible only for specific relations (e.g., left/right via horizontal displacement, inside/contains via box coverage) and abstains for others (e.g., front/behind, on/above) where 2D geometry is insufficient.

3. Training-Free Evidence Decision

The decision stage aggregates predictions from the available states using a fixed, deterministic rule:

Forward Decision: If Geometry is eligible and its layout score exceeds a threshold ( $\tau=0.5$ ), it takes priority. Otherwise, the system selects the visual state (Full or Marker) with the larger option-score gap (the difference between the top candidate and the runner-up).
Reciprocal Consistency Refinement: For protocols with exact inverse relations (e.g., left $\leftrightarrow$ right), SEER performs a reciprocal check. It swaps the subject and object roles (and the S/O markers) and re-evaluates the query. If exactly one state (Full or Marker) respects the inverse relation while the other does not, the system switches to the consistent state. This acts as a conflict-resolution mechanism only when an exact inverse exists.

Key Contributions

Query-Specific Evidence Interface: The paper formulates spatial answering as an evidence-interface problem, introducing target-relation-hidden self-grounding. This turns the queried pair into role-explicit visual evidence without VLM training or external detectors.
Reciprocal Consistency Refinement: An optional, protocol-specific refinement that uses exact subject/object exchange to obtain an observable consistency cue for conflicting evidence states, requiring no learned calibration or target labels.
Mechanism Isolation: Through matched counterfactuals (e.g., cropping without markers, swapping S/O identities, using oracle boxes), the authors isolate local refocus and role-explicit conditioning as the primary drivers of improvement, rather than simple cropping or additional text instructions.

Experimental Results

SEER was evaluated on frozen, image-disjoint test sets and development pools across multiple models (Qwen3-Instruct/Thinking, Qwen2.5, R1-Onevision, InternVL3.5) and datasets (GQA, Visual Genome, EmbSpatial, VSR, SpatialSense).

GQA-Train900 (Frozen Test): On a test set of 900 unique images frozen before scoring, SEER achieved a pooled gain of +3.94 over the Full-image baseline (95% CI [2.17, 5.72]). The gain remained positive under label-independent hash counterbalancing (+2.56).
Development Pools (GQA/VG): Across 41,544 evaluations, SEER improved all eight model/dataset combinations. The pooled gain was +3.99 over Full. The reciprocal consistency refinement contributed an additional +1.36 over the Forward decision.
EmbSpatial: On 2,434 filtered pair-relation questions, SEER improved Qwen3-Instruct, Qwen3-Thinking, and InternVL3.5 by +4.35, +5.09, and +11.79 respectively.
Mechanism Analysis:
- Local Refocus: Moving from Full to a cropped view (Crop) yielded a +1.25 gain.
- Role Conditioning: Adding explicit S/O markers to the crop (Self-Marker) yielded an additional +1.64 gain.
- Oracle Gap: Using ground-truth boxes (Oracle-Marker) provided a +6.17 gain over Self-Marker, indicating significant headroom for localization quality.
- Robustness: Gains persisted even when removing ambiguous "on" relations or repeated entity names, though performance varied by relation type (e.g., strong gains on "inside," mixed results on "right-of").
Limitations: The method does not transfer uniformly to binary truth judgments (e.g., VSR, SpatialSense), where reciprocal consistency is less effective. Performance is bounded by the quality of self-grounding; while IoU is imperfect, gains persist even with low overlap.

Significance and Claims

The paper claims that explicit evidence construction is the principal intervention for improving spatial reasoning in frozen VLMs. By making query-specific locations and roles jointly inspectable, SEER mitigates failures caused by attending to the wrong instance or relying on language priors.

Evidence over Reasoning: The results suggest that even advanced reasoning checkpoints benefit significantly from better visual evidence interfaces, establishing that "a spatial answer is only as good as its input evidence."
Training-Free Efficiency: SEER achieves these gains without training, using a fixed rule and a frozen VLM. While it incurs a computational cost (approx. 7.12x a single Full score), the authors note that adaptive gating can retain ~81% of the gain at ~2.57x cost.
Modesty: The authors explicitly state that SEER is an evidence interface, not a certificate of exact grounding. The improvements do not imply perfect instance recovery, and the method remains limited by the visibility of relations in local appearance and 2D layout. The reciprocal consistency refinement is described as a "smaller protocol-specific refinement" rather than a universal solution.

In conclusion, SEER demonstrates that controlling the visual evidence state—specifically through target-relation-hidden self-grounding and role-explicit conditioning—can significantly repair spatial relation failures in VLMs, offering a robust, training-free alternative to model retraining or external detection tools.