Technical Summary: STEMBIND

Problem Statement

Multimodal Large Language Models (MLLMs) frequently exhibit a specific failure mode in Abstract Visual Reasoning (AVR) tasks: they can correctly perceive visual elements and identify the underlying logical rules but fail to bind those rules to the correct candidate answer. Existing AVR benchmarks collapse perception, rule induction, and answer selection into a single binary signal (right/wrong), making it impossible to distinguish whether a model failed to see the image, failed to infer the rule, or failed to apply the rule to the specific instance. Consequently, current evaluations cannot isolate the "binding gap"—the disconnect between knowing a rule and executing it on a specific visual stem.

Methodology: STEMBIND Benchmark

To address this, the authors introduce STEMBIND, a shared-stem diagnostic benchmark designed to localize reasoning failures to specific sub-steps on the same visual evidence.

1. Shared-Stem Diagnostic Design

Unlike traditional benchmarks where sub-questions are posed on different images, STEMBIND probes the same visual stem with three aligned questions:

• Perception (P): "What is in the image?" (Verifying visual encoding).
• Rule (R): "What pattern governs it?" (Verifying rule induction).
• Full (F): "Which option completes it?" (The standard AVR task).

This design allows for the attribution of a final-answer error (F-wrong) to specific causes:

• Can't See: P is wrong.
• Can't Reason: P is right, but R is wrong.
• Can't Bind: P and R are right, but F is wrong.

2. Taxonomy and Scale

The benchmark contains 2,298 curated, knowledge-light stems across nine auditable visual operations, divided into two families:

• Rule-Induction (RI): Positional transformation, stylistic continuation, attribute binding, quantitative progression, and relational mapping.
• Visual-Processing (VP): Paper-folding, viewpoint synthesis, mental rotation, and visual closure.
  These stems are expanded into 19,533 tasks (P/R/F probes), with an average of ~6.5 perception probes per stem.

3. Process Annotation (S1–S4)

Each Full (F) item is annotated with four reasoning stages based on Sternberg's componential theory of induction:

• S1 Encode: Scene graph construction.
• S2 Infer: Rule set derivation.
• S3 Map: Cross-panel alignment (binding the rule to the instance).
• S4 Apply: Target prediction.

4. Evaluation Protocol

The authors employ a Stage-wise Stimulus Augmentation (SSA) protocol. This involves cumulatively injecting verified stage content (e.g., providing the correct S1 or S2 output to the model) to test which specific stage's information repairs a failed item. This helps localize the dominant failure point behaviorally.

Key Results

The authors evaluated 24 frontier MLLM configurations (including proprietary models like GPT-5.4, Claude-Opus-4.7, and open-source families like Qwen3.5, InternVL3.5, and Gemma 4). The evaluation yielded four primary findings:

1. The R–F Chasm

Rule accuracy (R) significantly exceeds Full-item accuracy (F) on 22 of the 24 models. The gap often ranges from 20 to 34 percentage points. For instance, on Qwen3.5-27B, R accuracy is 65.4% while F accuracy is only 41.3%. This indicates that most failures occur after the rule has been successfully identified.

2. The Persistent Binding Gap

Even on the strict subset of stems where models correctly answer both Perception (P) and Rule (R), they still answer the Full item (F) incorrectly 51.2% of the time. This "Binding Gap" (BG = 0.51) persists across model families and sizes, suggesting a fundamental inability to map inferred rules to specific visual candidates.

3. The S3 Bottleneck

Process diagnostics and SSA interventions localize the dominant failure to S3 (Map).

• Stage-wise Judging: S3 accuracy is consistently the lowest among the four stages (mean StepAcc ~30.5% for Qwen3.5), significantly lower than S1, S2, and S4.
• SSA Gains: The largest performance gains occur when S3 alignment information is injected (H2→H3 transition), confirming that the "rule-to-instance mapping" step is the primary bottleneck.

4. Scaling and Thinking Do Not Help

• Scaling: Increasing model size (e.g., within the Qwen3.5 family) improves Perception and Rule accuracy but does not reliably close the F-side binding gap. In some cases (e.g., Qwen3.5-122B MoE), scaling leads to a collapse in Rule accuracy.
• Thinking Mode: Enabling explicit "thinking" modes (Chain-of-Thought) generally lowers both Rule and Full-item accuracy across all tested models, even though it often improves Perception. Longer reasoning traces appear to disrupt the slot-level correspondence required for binding.

Significance and Contributions

The paper reframes AVR evaluation from a "final-answer ranking" to a "diagnostic instrument" that locates where abstract visual reasoning breaks down.

1. Diagnostic Instrument: STEMBIND is the first benchmark to probe perception, rule, and full-item solving on the same visual stem, enabling the isolation of the binding failure mode.
2. Process-Aware Attribution: By combining shared-stem probes with S1–S4 annotations and SSA interventions, the authors provide a protocol to attribute errors to specific sub-steps rather than treating the model as a black box.
3. Empirical Diagnosis: The work identifies rule-to-instance binding as the concrete, dominant failure mode for current frontier MLLMs, distinct from perception or rule induction deficits.
4. Evidence Against Common Fixes: The study provides empirical evidence that simply scaling model parameters or enabling explicit thinking modes does not resolve the binding gap, suggesting that future progress in vision-grounded reasoning requires targeted architectural or training interventions for the mapping stage.

The authors conclude that the primary challenge for MLLMs in AVR is not seeing the image or finding the rule, but binding the known rule to the correct visual instance.