💻 computer science

Explainable and Resource-Efficient Spatial Reasoning in Multimodal LLMs for Decision-Critical Applications

This paper introduces ByDeWay-V2, a training-free framework that enhances spatial reasoning and explainability in Multimodal Large Language Models for decision-critical applications by integrating explicit, human-readable pairwise geometric relations alongside depth cues, thereby significantly reducing hallucinations and improving performance on spatial benchmarks while operating efficiently within strict resource constraints.

Original authors: Piyush Jain, Kousik Dasgupta, Rajarshi Roy, Subarna Tripathi

Published 2026-07-30

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Piyush Jain, Kousik Dasgupta, Rajarshi Roy, Subarna Tripathi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot how to see the world. This robot isn't just a camera; it's a "Multimodal Large Language Model" (or MLLM for short), a type of artificial intelligence that can look at a picture and talk about it, answer questions, and even make decisions. Think of it as a brilliant student who has read every book in the library but has never actually walked outside. Because it learned mostly from text, it sometimes gets the physical world wrong. It might confidently tell you that a cat is sitting on a table when, in the photo, the cat is actually under the table. This mistake is called a "hallucination."

In the real world, these mistakes can be dangerous. If a robot is supposed to pick up a cup without knocking it over, or if a safety system needs to spot a person standing in front of a moving car, being "sort of right" isn't good enough. We need the robot to be precise and, just as importantly, we need to know why it made that decision. Can we trust it? The big question scientists are asking is: How do we stop these AI models from making up facts about space and position, and how do we make their thinking clear enough for humans to check?

This is where a new paper comes in with a clever, low-cost solution called ByDeWay-V2. The researchers realized that previous attempts to fix these robots were a bit like giving them a blurry map. They knew roughly how far away things were (like "close," "medium," or "far"), but they didn't know exactly where one object was in relation to another. It's like knowing a person and a refrigerator are both in the "closest" part of the room, but not knowing if the person is standing next to the fridge or inside it.

To solve this, the team built a system that acts like a super-organized tour guide for the robot's eyes. Before the robot tries to answer a question, the system first snaps a quick photo of the scene and uses a specialized tool (called YOLO-World-L) to draw invisible boxes around every object it sees. Then, it does some quick math to figure out the exact geometry: "The cup is 5 inches to the left of the laptop," or "The cat is touching the vase." It turns these math facts into simple, human-readable sentences and feeds them directly to the robot as a "cheat sheet."

The results are quite impressive. When they tested this new method on different AI models, the improvement was dramatic. For a smaller, lighter model called BLIP-Base, the system turned a near-random guess (scoring a 0.05) into a solid, competitive performance (scoring 0.53). On a tricky test called BLINK, which specifically asks about where things are relative to each other, the system helped a top-tier model improve its score by 46%. Perhaps most exciting for real-world use, this whole process doesn't require retraining the robot or using massive supercomputers. The lightest version of their system can run on a standard computer processor (CPU) using less than 40 words of "memory" (tokens), proving that you don't need a giant brain to have a clear, trustworthy understanding of space.

In short, ByDeWay-V2 doesn't just make the robot smarter; it makes the robot honest. By giving it explicit, step-by-step evidence about where objects are, the robot stops guessing and starts explaining its reasoning, making it much safer and more reliable for the critical jobs we want it to do.

Technical Summary: Explainable and Resource-Efficient Spatial Reasoning in MLLMs for Decision-Critical Applications

Problem Statement

Multimodal Large Language Models (MLLMs) are increasingly deployed in decision-critical pipelines such as robotics, embodied AI, and safety monitoring. However, their reliability is hindered by two persistent challenges: visual hallucinations (generating content not grounded in visual input) and spatial reasoning deficits (struggling to interpret geometric, projective, and topological relationships between objects).

While previous work, ByDeWay, introduced Layered-Depth-Based Prompting (LDP) to mitigate hallucinations using monocular depth estimations, it operates at a macro-regional level. LDP fails to resolve fine-grained spatial interactions between objects residing in the same geometric plane (e.g., determining if an object is "left of," "inside," or "touching" another). This limitation is critical in decision-support settings where operators require auditable, human-readable justifications for model judgments, not just black-box outputs. Existing training-free methods often improve accuracy without providing intermediate, inspectable evidence.

Methodology: ByDeWay-V2

The authors propose ByDeWay-V2, a fully training-free framework that enhances the original LDP pipeline by integrating explicit spatial relational context. The system operates via two parallel branches to process an input image, synthesizing the outputs into a structured prompt for the downstream MLLM:

Depth-Aware Branch (LDP):
- Utilizes DepthAnything V2 to generate an affine-invariant inverse depth map.
- Partitions the scene into three depth layers (closest, mid-range, farthest) using percentile thresholds.
- Employs KOSMOS-2 to generate natural language captions for each layer, providing coarse 3D depth organization.
SpatialAnalyser Branch (New Module):
- Uses YOLO-World-L, an open-vocabulary real-time detector, to identify and localize all relevant objects in the scene with precise bounding boxes.
- Computes pairwise geometric heuristics between detected objects, including:
  - Vertical/Horizontal relations: Based on center-point coordinates (e.g., "left of," "above").
  - Containment and Contact: Calculated via Intersection-over-Union (IoU) ratios (e.g., "inside," "touching").
  - Proximity: Based on normalized Euclidean distance between centers.
- Resolves these relations in a priority order (containment > contact > vertical/horizontal > proximity) and serializes them into human-readable predicates (e.g., "The cup is left of the laptop").
Hybrid Prompt Construction:
- The final prompt combines the depth-layer captions ( $C_{LDP}$ ), the explicit spatial predicates ( $C_{Spatial}$ ), and the user query ( $Q$ ).
- This approach bridges the gap between 3D scene depth and 2D inter-object spatial semantics without requiring any parameter updates to the MLLM.

Key Contributions

Enhanced Framework: Introduces ByDeWay-V2, a training-free extension of LDP that incorporates a dedicated SpatialAnalyser module for explicit spatial relation extraction.
Hybrid Prompting: Presents a methodology that combines coarse depth-aware scene decomposition with precise, bounding-box-derived pairwise geometric predicates.
Modularity: Demonstrates a model-agnostic approach that requires no fine-tuning of downstream MLLMs.
Resource Efficiency: The framework is designed to operate under strict token budgets (e.g., a 40-token limit on CPU for ViLT), making it suitable for resource-constrained, real-time decision-support settings.

Experimental Results

The framework was evaluated on three benchmarks (VSR, BLINK, POPE) across three MLLMs: Qwen2.5-VL-7B, BLIP-Base, and ViLT-B/32.

BLINK Spatial Subset: ByDeWay-V2 achieved a 46% relative improvement in F1 score over LDP for Qwen2.5-VL (rising from 0.496 to 0.727). This benchmark specifically targets fine-grained inter-object queries ("is object A left of object B?"), where the explicit predicates provide direct evidence.
VSR Benchmark: For the lightweight BLIP-Base, the addition of spatial context recovered performance from near-random levels (F1 0.053) to a competitive F1 of 0.525. For ViLT-B/32, substituting verbose depth captions with compact spatial predicates increased precision from 0.549 to 0.575, demonstrating token efficiency.
POPE Benchmark (Hallucination Reduction): ByDeWay-V2 consistently improved hallucination resistance. BLIP-Base saw overall accuracy rise from 86.00% (LDP) to 90.67% (LDP+Spatial), with an F1 increase from 0.873 to 0.919.
Per-Category Analysis: The most significant gains were observed in proximity and projective relations (e.g., "near," "left of"), where F1 scores for BLIP-Base improved by over +0.50, confirming that geometric grounding effectively anchors MLLM judgments.

Significance and Claims

The paper claims that ByDeWay-V2 successfully addresses the "spatial reasoning gap" where depth-only methods fail to resolve fine-grained object interactions. By injecting explicit, auditable spatial predicates, the framework not only improves accuracy but also provides the explainability required for decision-critical applications.

The authors emphasize that this approach is particularly effective for compact models operating under strict computational constraints. While they note a precision-recall trade-off in Qwen2.5-VL (attributed to the model's "Vision-First" instruction overriding context), the results confirm that explicit geometric grounding reliably reduces hallucinations and improves spatial reasoning without the need for expensive training data or parameter updates. The work positions itself as a practical, immediate solution for enhancing the trustworthiness of MLLMs in safety-critical domains.