Can VLMs Reason Robustly? A Neuro-Symbolic Investigation

This paper demonstrates that while standard Vision-Language Models fail to generalize under covariate shifts in visual deductive reasoning tasks, the proposed VLC method achieves robust performance by decoupling perception from reasoning through a neuro-symbolic framework that combines VLM-based concept recognition with exact symbolic circuit execution.

The Gist

Imagine you have a brilliant student named VLM (Vision-Language Model). This student is amazing at looking at a picture and describing what they see. If you show them a photo of a cat, they'll say, "That's a fluffy cat!" If you show them a messy room, they'll list the toys and clothes. They are great at perception.

But here's the problem: When you ask this student to solve a logic puzzle based on that picture, they start to stumble, especially if the puzzle looks slightly different from the ones they practiced on.

This paper investigates why that happens and proposes a new way to teach these AI students how to reason properly.

The Problem: The "Rote Learner" vs. The "True Thinker"

The researchers tested the VLM on three types of logic puzzles:

1. Adding numbers written in a picture.
2. Calculating a "XOR" (a logic game where you flip bits) based on a sequence of digits.
3. Checking rules about shapes and colors (e.g., "Do all the red circles have to be big?").

They found a frustrating pattern:

• The "Cramming" Method: When they trained the VLM by showing it thousands of examples and letting it guess the answer (a method called end-to-end fine-tuning), the student got 99% on the practice tests.
• The "New Test" Disaster: But when the researchers changed the test slightly—like asking the student to add 7 numbers instead of 3, or use different fonts—the student's score plummeted to near zero.

The Analogy: Imagine the student didn't actually learn how to add. Instead, they memorized the specific look of the answer keys for 3-digit problems. When you gave them a 7-digit problem, they panicked because the "pattern" they memorized didn't fit. They were cramming, not reasoning.

The Failed Fixes: "The Black Box"

The researchers tried two popular modern fixes:

1. Prism: They let the VLM describe the picture, then handed the description to a super-smart text AI (an LLM) to solve the math.
   • Result: The text AI was good at simple math but terrible at complex logic. It was like hiring a math genius who is bad at following strict logical rules.
2. ViperGPT: They asked the AI to write a computer program to solve the problem, then ran that program.
   • Result: This worked great if the program was perfect. But if the program made a tiny mistake (like misidentifying a shape), the whole answer was wrong. It was like building a Rube Goldberg machine; if one gear slips, the whole thing fails.

The Solution: VLC (The "Human + Calculator" Team)

The researchers proposed a new system called VLC. Instead of trying to make one giant AI brain do everything, they split the job into two distinct roles, like a Detective and a Calculator.

Phase 1: The Detective (The VLM)

The VLM looks at the image and simply answers: "What do I see?"

• "I see a 6, a 4, and a 0."
• "I see a red square and a blue circle."
• It doesn't try to solve the puzzle yet. It just gathers the facts.

Phase 2: The Calculator (The Circuit)

This is the magic part. The researchers take the rules of the puzzle (e.g., "Add these numbers" or "Check if colors match") and write them into a rigid, unchangeable computer circuit.

• This circuit is like a mechanical calculator or a flowchart. It cannot "guess." It cannot "hallucinate." It follows the rules exactly, step-by-step.
• The Detective's facts are fed into this Calculator.
• The Calculator crunches the numbers and spits out the answer.

The Analogy:
Imagine you are taking a math test.

• Old Way: You try to memorize the answers to every possible test. If the teacher changes the numbers, you fail.
• VLC Way: You have a friend (the VLM) who is great at reading the numbers on the paper and telling you what they are. Then, you hand those numbers to a robot calculator (the Circuit) that you built yourself. The robot knows the rules of addition perfectly. Even if the numbers are huge or the font is weird, the robot will always get the right answer because it follows the rules, not guesses.

Why This Matters

The paper shows that VLC is incredibly robust.

• It didn't matter if the test had 3 objects or 7 objects.
• It didn't matter if the objects were handwritten or printed.
• As long as the "Detective" could see the objects clearly, the "Calculator" could solve the logic puzzle perfectly.

The Big Takeaway:
To make AI truly smart at reasoning, we shouldn't just throw more data at a giant brain and hope it figures it out. Instead, we should decouple the tasks:

1. Use AI for what it's good at: Seeing and recognizing things.
2. Use strict, symbolic code for what it's bad at: Following logical rules.

By combining the two, we get a system that doesn't just "cram" for the test but actually understands how to think, making it reliable even when the world changes around it.

Technical Summary

1. Problem Statement

The paper investigates the robustness of Vision-Language Models (VLMs) under covariate shifts. Specifically, it addresses scenarios where the distribution of perceptual inputs changes (e.g., the number of objects in an image increases), while the underlying logical rules for prediction remain constant.

• Core Question: Can VLMs, particularly those fine-tuned via gradient-based end-to-end training, learn the underlying reasoning function and generalize to out-of-distribution (OOD) data where the input complexity varies?
• Context: While VLMs excel at recognition tasks, their ability to perform structured, rule-based deductive reasoning is unclear. The authors hypothesize that standard fine-tuning may cause models to memorize statistical correlations in the training data rather than learning the true logical function, leading to failure under distribution shifts.

2. Methodology

A. Experimental Setup: Visual Deductive Reasoning Tasks

The authors utilize the rsbench benchmark suite to generate three distinct visual deductive reasoning tasks. In these tasks, rules are explicitly provided (unlike standard VQA), and the test set involves a covariate shift (more objects than the training set).

1. MNAdd: Arithmetic addition of two multi-digit numbers (handwritten MNIST digits).
2. MNLogic: Logical XOR of a sequence of binary digits.
3. KandLogic: Relational check to determine if all objects of the same shape share the same color.

B. Baselines Evaluated

The study compares several reasoning paradigms:

• End-to-End (E2E) Reasoning: Pretrained VLMs prompted directly.
• End-to-End Fine-Tuning: VLMs fine-tuned on the smallest dataset (fewest objects) using causal language modeling loss.
• Prism: A neuro-symbolic approach where a VLM recognizes concepts, and a Large Language Model (LLM) performs the reasoning.
• ViperGPT: An approach where an LLM generates executable Python code to decompose the task, calling various pretrained models (detection, VLM) to execute steps.

C. Proposed Method: VLC (Vision-Language Circuit)

The authors propose VLC, a neuro-symbolic framework that strictly decouples perception from reasoning into two sequential phases:

1. Phase I: VLM-based Concept Recognition:
   • A VLM (e.g., Qwen2.5-VL) is prompted to identify object concepts (digits, shapes, colors) from the image.
   • In-context learning (few-shot examples) is used to ensure the output follows a structured format.
   • The textual output is parsed and converted into binary inputs.
2. Phase II: Circuit-based Symbolic Reasoning:
   • The task rules are compiled offline into a Symbolic Decision Diagram (SDD), a type of Boolean circuit.
   • The circuit acts as a deterministic mapper. It takes the binary inputs from Phase I and executes the logical rules exactly (e.g., performing bitwise addition or XOR).
   • The circuit output is converted back to the final answer.
   • Key Distinction: Unlike other neuro-symbolic methods, VLC does not fine-tune the circuit or the VLM jointly. The reasoning logic is hard-coded into the circuit, ensuring exact rule execution.

3. Key Contributions

1. Empirical Evidence of Failure: Demonstrated that VLMs fine-tuned end-to-end achieve high in-distribution accuracy but fail catastrophically on OOD data (e.g., dropping from ~99% to ~50% when object counts increase), proving they do not learn the underlying reasoning function.
2. Limitations of Black-Box Neuro-Symbolic Approaches: Showed that approaches like Prism (LLM reasoning) and ViperGPT (code generation) exhibit inconsistent robustness. Prism struggles with logical functions (XOR) despite good arithmetic performance, and ViperGPT is highly sensitive to errors in intermediate steps (e.g., object detection failures).
3. Proposal of VLC: Introduced a simple, two-stage neuro-symbolic pipeline that compiles rules into circuits. This approach guarantees that if the perception is correct, the reasoning is logically sound and robust to input distribution shifts.
4. Scaling Analysis: Found that scaling VLM size improves concept recognition but does not necessarily improve reasoning capabilities for specific logical functions, reinforcing the need for explicit symbolic integration.

4. Experimental Results

• Robustness under Covariate Shift:
  • End-to-End Fine-tuning: Failed to generalize. For example, on MNLogic, accuracy dropped from 99.83% (3 objects) to 51.95% (7 objects).
  • Prism & ViperGPT: Showed mixed results. Prism improved on arithmetic but degraded on logic tasks. ViperGPT suffered from error propagation (e.g., detection failures on MNAdd led to near-zero accuracy).
  • VLC: Consistently achieved high accuracy across all object counts. On the 7-object/7-digit test sets, VLC maintained strong performance (e.g., 60.31% on MNLogic vs. 51.95% for fine-tuned VLM), significantly outperforming baselines.
• Ablation Studies:
  • Fine-tuning for Recognition: Fine-tuning the VLM specifically for concept recognition (not reasoning) significantly boosted VLC's performance, confirming that the bottleneck in VLC is perception, not reasoning.
  • Model Scaling: Increasing VLM size improved concept recognition accuracy but had diminishing returns on reasoning tasks for end-to-end models. Increasing LLM size in Prism improved arithmetic reasoning but had little effect on logical XOR tasks.
• Gap Analysis: In Prism, a large gap exists between concept accuracy and task accuracy (due to LLM reasoning errors). In VLC, task accuracy closely matches concept accuracy, proving the reasoning module is reliable.

5. Significance and Conclusion

The paper provides a critical insight into the limitations of current VLMs: gradient-based fine-tuning is insufficient for learning robust, generalizable reasoning functions.

• Theoretical Implication: It suggests that for tasks requiring strict logical adherence, the reasoning function should be explicitly encoded (via symbolic programs/circuits) rather than learned implicitly through weights.
• Practical Impact: The VLC framework offers a path toward interpretable and robust AI by combining the perceptual strengths of deep learning with the logical guarantees of symbolic AI. It demonstrates that a simple "perception + symbolic execution" pipeline can outperform complex end-to-end or black-box neuro-symbolic systems in scenarios involving distribution shifts.
• Future Directions: The authors note limitations, such as the need to automatically extract symbolic rules from natural language and the dependency of the system on the VLM's initial recognition accuracy.

In summary, the paper argues that to achieve robust reasoning, we must move away from treating reasoning as a black-box learning problem and instead adopt a neuro-symbolic approach where the logic is explicitly defined and executed.