Technical Summary: Ill-Posed by Design: Probing Evidence Use in VLMs

Problem Statement

Counterfactual analysis is a standard tool for studying evidence use in Vision-Language Models (VLMs), yet its diagnostic value is limited on well-posed tasks. In tasks like object recognition, multiple visual cues often independently support the same answer; masking one cue may leave the prediction unchanged, making it impossible to distinguish between a cue being unused versus merely redundant.

This paper proposes monocular metric object-size estimation as an ill-posed diagnostic setting. Because physical scale cannot be determined from a single uncalibrated image without stereo cues or known references, the task is fundamentally underdetermined. Models must rely on imperfect, potentially conflicting cues such as category priors, target appearance, local context, apparent image size, and scene geometry. Standard accuracy metrics collapse these strategies into a single score, obscuring which specific evidence channel a model relies on or fails to use.

Methodology

Benchmark: Metric VQA

The authors introduce Metric VQA, a benchmark comprising 10,813 dimension queries paired with ground-truth measurements, split into two complementary datasets:

1. Objectron-derived Split (10,482 queries): Curated frames from the Objectron dataset with 3D bounding box ground truth, covering nine everyday object categories.
2. In-the-Wild (ITW) Split (331 queries): Cluttered, real-world scenes with tape-measured ground truth, containing multiple objects with arbitrary occlusion and lighting.

Counterfactual Interventions

The core methodology involves applying six image-level interventions while holding the natural language query fixed. These interventions probe six distinct evidence channels:

1. Text-only: Removing the image modality entirely (or replacing it with a gray image) to measure reliance on language priors.
2. Target Pixels: Masking the entire scene except for the target object (object_only).
3. Target Identity: Swapping the target object with a different object in the same scene (wrong_object) while preserving location and scene statistics.
4. Local Context: Keeping the target and a thin ring of surrounding context (object_ring).
5. Apparent Size: Rescaling the target object within the image (zoom-in 1.4× and zoom-out 0.6×) to test sensitivity to pixel footprint.
6. Global Scene Geometry: Applying radial lens distortion (barrel and pincushion) to perturb vanishing-point structures while preserving object identity.

Evaluation Setup

The study evaluates 12 open-weight VLMs ranging from 3B to 397B parameters (including Qwen, InternVL, LLaVA, and Molmo families). Performance is measured against:

• A text-only frontier LLM (ChatGPT 5.5) queried without images.
• A human baseline on the ITW split.
• Acc@10% (fraction of predictions within 10% of ground truth) is the primary metric.

The authors also analyze LoRA fine-tuning as an actionable intervention, training adapters on the Objectron split to observe how supervised learning alters evidence selection patterns.

Key Results

1. Language Priors Dominate in the Wild

Even the largest open VLMs tested (Qwen3.5-397B, InternVL3.5-241B) trail a text-only frontier LLM on the in-the-wild split of Metric VQA. This indicates that for metric estimation, strong category-level language priors often explain more performance variance than visual evidence in unstructured environments.

2. Stratified Evidence Use

Counterfactual signatures reveal a hierarchical pattern of evidence reliance:

• Target Identity is Load-Bearing: Replacing the target object causes the most consistent and significant accuracy drops across models, indicating that models condition estimates heavily on what object is visible.
• Target Pixels and Local Context are Selective: Only a subset of larger models leverages raw target pixels for metric estimation. Local context provides a smaller, model-dependent benefit, and for some models, scene clutter acts as a distractor rather than a cue.
• Apparent Size is Non-Directional: While predictions shift when object size changes, the shifts are not directionally consistent (e.g., zooming in does not systematically increase predicted size). Apparent size acts as a destabilizing signal for many models rather than a reliable metric cue.
• Scene Geometry is Largely Unused: Global scene geometry (perspective/vanishing points) is the weakest channel; only one model showed a reliable accuracy drop under lens distortion, and no model exhibited directionally consistent behavior.

3. LoRA Fine-Tuning Limitations

Supervised LoRA fine-tuning improves overall accuracy on the metric estimation task. However, the analysis shows that models do not learn to leverage scene geometry or develop new visual-geometry understanding. Instead, fine-tuning improves the specific channels the base model already partially uses (e.g., stabilizing language priors, deepening target-identity sensitivity, or utilizing local context). The task is learnable, but the mechanism is patching existing weak channels rather than acquiring new geometric capabilities.

Significance and Claims

The paper argues that ill-posed tasks serve as a superior diagnostic tool for multimodal systems compared to well-posed benchmarks. By forcing models to choose among imperfect, non-redundant evidence sources, counterfactual interventions isolate specific cue usage patterns that aggregate accuracy metrics hide.

Key Takeaways:

• Diagnostic over Ranking: The study prioritizes understanding how models reason (evidence selection) over simply ranking them by accuracy.
• The Illusion of Visual Measurement: A VLM's centimeter estimate should not be treated as a reliable measurement. It is better understood as a category-level prior with selective, model-dependent visual modulation.
• Limitations of Current Training: Standard next-token-loss fine-tuning (LoRA) fails to induce genuine scene-geometry understanding, suggesting that current training paradigms may not be sufficient to bridge the gap between language priors and physical reasoning.

The authors release Metric VQA (data, intervention variants, predictions, and code) to support reproducible diagnostic analysis of evidence use in future multimodal systems.