🤖 machine learning

Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs

This paper introduces Perception Programs (P $^2$ ), a training-free method that converts dense, pixel-level vision tool outputs into compact, language-native summaries, enabling multimodal language models to achieve significant performance gains on visual reasoning tasks without requiring model retraining or modifications.

Original authors: Muhammad Kamran Janjua, Hugo Silva, Di Niu, Bahador Rashidi

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muhammad Kamran Janjua, Hugo Silva, Di Niu, Bahador Rashidi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant detective (a Multimodal Language Model, or MLLM) who is amazing at solving mysteries using words, logic, and stories. However, this detective has a strange weakness: they are terrible at reading raw data like spreadsheets, blueprints, or pixel-by-pixel maps.

If you hand the detective a 100-page spreadsheet of numbers representing a video's motion, they get overwhelmed. They might guess the answer based on their general knowledge ("Oh, videos usually move right?") rather than actually seeing the motion. This is the problem the paper addresses: AI models are great at language but bad at "reading" raw visual data.

The Problem: The "Pixel Soup"

The researchers tried giving the detective tools to see better. They handed them a "depth map" (a map showing how far things are) or an "optical flow map" (a map showing how things move).

But here's the catch: these tools output dense, pixel-level data. It's like handing the detective a bowl of soup made of millions of tiny, uncooked rice grains (pixels) and asking them to find the specific grain that tells the story. The detective just stares at the soup, gets confused, and ignores the clues because they don't speak "Rice Grain."

The Solution: "Perception Programs" (P2)

The authors introduced a new method called Perception Programs (P2). Think of P2 as a super-efficient translator or a smart sous-chef.

Instead of handing the detective the raw bowl of pixel-soup, the sous-chef (P2) takes that soup, cooks it down, and serves the detective a crisp, clear sentence on a notepad.

Raw Tool Output: "Here is a 100x100 grid of numbers. 45 of them are 0.2, 30 are 0.8..." (Confusing soup).
Perception Program (P2): "The camera is moving Left. 90% of the scene shifted leftward." (Clear sentence).

The detective (the AI model) can now read this sentence instantly and solve the mystery with high accuracy. They don't need to be retrained; they just need the information presented in a language they understand.

How It Works in Real Life (The Analogies)

The "Left vs. Right" Camera Mystery:
- Scenario: You show the AI two frames of a video and ask, "Is the camera moving left or right?"
- Old Way (Raw Tool): The AI gets a massive list of numbers for every single pixel. It gets lost in the noise and guesses wrong.
- P2 Way: The tool analyzes the pixels and says, "Hey, 93 out of 100 patches of the image moved Left." The AI reads this, nods, and says, "Okay, the answer is Left."
The "Jigsaw Puzzle" Analogy:
- Scenario: You have a puzzle with a missing corner and two candidate pieces. Which one fits?
- Old Way: The AI tries to compare the millions of pixels in the missing corner with the candidate pieces. It's slow and error-prone.
- P2 Way: The tool calculates the "similarity score" and tells the AI: "Piece A matches the edge with 95% confidence; Piece B matches with 40%." The AI just picks Piece A.

Why This Is a Big Deal

The paper shows that you don't need to build bigger, more expensive AI models or train them for years to make them smarter at seeing. You just need to change the format of the information you give them.

No Training Needed: You can take any existing AI model (even small, open-source ones) and plug this "translator" (P2) in, and it instantly becomes a visual reasoning expert.
Massive Gains: In tests, this simple translation trick boosted the accuracy of some models by 20% to 25%. It turned a model that was barely passing a test into one that was acing it.
Efficiency: It also saves money and time. The AI doesn't have to waste energy trying to "read" millions of pixels; it just reads a short summary.

The Bottom Line

The paper's main message is: "Don't show the AI the pixels; show it the clues."

By converting complex visual data into simple, structured language summaries (Perception Programs), we unlock the AI's ability to reason about the visual world without needing to rebuild the AI itself. It's like giving a chef a recipe written in their native language instead of a list of raw ingredients in a foreign code. Suddenly, they can cook a masterpiece.

1. Problem Statement

Multimodal Large Language Models (MLLMs) are increasingly paired with specialized vision tools (e.g., depth estimation, optical flow, visual correspondence) to enhance visual reasoning. However, a critical bottleneck exists: MLLMs fail to fully utilize the rich information provided by these tools.

The Misalignment: When raw tool outputs (dense, pixel-level data like depth maps or flow fields) are fed directly into MLLMs, they appear as low-level visual tokens. This format misaligns with the language-native reasoning substrate of LLMs.
The Consequence: MLLMs tend to ignore the dense visual signals, relying instead on language priors or making errors in interpreting spatial relationships. Even with accurate tool surrogates, models often fail to recover relative ordering (e.g., near-to-far depth) or global motion direction.
The Question: Is the bottleneck the number of tool calls or model size? The authors argue it is the representation of the visual information itself.

2. Methodology: Perception Programs (P2)

The authors propose Perception Programs (P2), a training-free, model-agnostic framework that rewrites raw tool outputs into compact, structured, language-native summaries.

Core Concept

Instead of feeding pixels or dense arrays, P2 converts tool outputs into a symbolic, YAML-like text block that describes:

What is present (the perceptual signal).
Where it is grounded spatially in the image.
How local regions relate (optional semantic relations).

Construction Algorithm

Primitives & Grids: The image domain is partitioned into a grid (e.g., $P \times P$ ). Each cell is a primitive $p$ with a normalized spatial coordinate $c_p \in [0, 1000]^2$ .
Read-outs ( $r_p$ ): For each primitive, the tool's raw data is summarized into a scalar or categorical value.
- Depth: Min/Max depth values in the patch.
- Optical Flow: Categorical direction ('left' or 'right') based on mean horizontal flow.
- Correspondence: Normalized coordinates of matched keypoints.
- Object Detection: Confidence scores and category labels.
Relations ( $T$ ): Optional symbolic triples (e.g., (patch_a, "in-front-of", patch_b)) are generated by comparing statistics between primitives to capture spatial relationships.
Serialization: The structured data is serialized into a text block (e.g., YAML) that an MLLM can parse and reason over using its native language capabilities.

Modality Instantiations

The paper demonstrates P2 across six modalities:

Depth: Converts depth maps into min/max ranges per grid cell with relative ordering relations.
Optical Flow: Converts vector fields into discrete motion directions ('left'/'right').
Visual/Semantic Correspondence: Maps keypoints to normalized coordinates and similarity scores.
Jigsaw: Uses structural/color similarity scores for edge matching.
Object Localization: Uses bounding box coordinates and confidence scores.

3. Key Contributions

Unified Representation: Introduced P2, a symbolic framework that generalizes across diverse perception tasks (depth, flow, correspondence, etc.) without requiring model fine-tuning or architectural changes.
Training-Free & Plug-and-Play: P2 requires no parameter updates to the MLLM. It acts as a middleware that converts existing tool outputs into a format any off-the-shelf MLLM can reason over.
Comprehensive Evaluation: Validated on the BLINK benchmark (6 perception-centric sub-tasks), showing consistent and massive improvements over both standard MLLMs and raw tool-augmented baselines.
Bottleneck Analysis: Demonstrated that the primary limitation in tool-augmented MLLMs is not the lack of tools, but the representation mismatch between pixel-level data and linguistic reasoning.

4. Experimental Results

The authors evaluated P2 on frontier models (GPT-5 Mini, Gemini 2.5 Pro) and open-source models (InternVL3.5, Qwen3VL) across six BLINK tasks.

Performance Gains:
- GPT-5 Mini: Accuracy on Multi-View Reasoning jumped from 41.35% (Standard) to 86.47% (P2). Relative Depth improved from 52.42% to 81.45%.
- Open-Source Models: InternVL3.5-4B and Qwen3VL-4B saw absolute gains of 21–25% over image-based tool variants, surpassing prior state-of-the-art (SOTA) methods that required extensive training or reinforcement learning.
- Overall: P2 achieved a 19.66% average improvement across all tasks and models, setting new SOTA results.
Comparison to Baselines:
- P2 outperformed complex agentic methods (e.g., Visual Sketchpad, MMFactory) and RL-based approaches (e.g., ReVPT, ViGoRL).
- Notably, P2 achieved these results with significantly fewer tokens per sample compared to methods that generate code or long chains of thought.
Analysis of Failure Modes:
- Without P2, models failed to preserve relative depth ordering as grid resolution increased (Kendall's $\tau$ approached zero).
- With P2, models successfully reasoned over the summarized cues, even on smaller models.

5. Significance and Impact

Paradigm Shift: The paper challenges the prevailing notion that better visual reasoning requires larger models or more complex tool-calling pipelines. Instead, it argues that better representation of existing tool signals is the key.
Efficiency: By converting dense data into language-native summaries, P2 reduces the cognitive load on the MLLM, allowing it to focus on high-level reasoning rather than low-level pixel interpretation.
Accessibility: Since P2 is training-free and model-agnostic, it democratizes high-performance visual reasoning, enabling smaller, open-source models to compete with or surpass massive proprietary models in perception tasks.
Future Direction: Suggests a move toward "principled interfaces" between perception and language, where the format of data transfer is as critical as the data itself.

In conclusion, Perception Programs (P2) bridge the gap between computer vision tools and language models by translating "pixels" into "cues," unlocking the full reasoning potential of MLLMs without the need for additional training or model modifications.