Technical Summary: ROSE – Benchmarking the Perception-to-Action Gap in Multimodal Models

Problem Statement

While Multimodal Large Language Models (MLLMs) have demonstrated significant progress in visual perception, reasoning, and generation, a critical gap remains in their ability to translate visual evidence into context-dependent actions. Existing benchmarks often aggregate heterogeneous tasks, evaluate fixed questions per image, or couple visual decisions with complex planning and environment interaction. Consequently, they fail to isolate a specific capability: Can an MLLM convert a consistent visual interpretation into the exact action required by a changing task context, while the underlying scene remains fixed?

Current models often succeed in recognizing "what is present" (perception) but struggle to decide "how to act" when the relevant region or required symbolic output changes. This paper posits that a correct action is not merely a by-product of recognition but a context-dependent commitment, and that the transition from perception to action is currently a significant bottleneck for state-of-the-art models.

Methodology: The ROSE Benchmark

To address this, the authors introduce ROSE (Reference-conditioned Oddity and Symbolic Execution), a controlled benchmark designed to hold visual evidence fixed while varying region constraints and required symbolic outputs.

Benchmark Design Principles

• Scene Coupling: ROSE derives multiple tasks from a single visual scene. The visual evidence (a grid of elements) and the global set of "exception" cells (deviations from a majority reference) remain constant.
• Implicit Majority Reference: Scenes contain no explicit target names. The model must infer the majority reference to identify sparse exceptions.
• Controlled Variation: The benchmark intervenes only on the relevant context (e.g., a specific row, column, or visual region) and the required output operation (e.g., counting vs. clicking coordinates).

Task Suite

ROSE consists of five coupled task templates derived from the same scene:

1. Global Counting (G-Cnt): Count all exception cells in the full grid.
2. Local Counting (L-Cnt): Count exception cells within a numerically specified region.
3. Local Clicking (L-Clk): Return the exact coordinates of exception cells within a numerically specified region.
4. Visual-Region Clicking (V-Clk): Return exact coordinates of exceptions within a region visually highlighted in the image.
5. Exclusion Clicking with Count Submission (Excl-CS): Click exceptions outside a specified region and submit the count, testing consistency between action and cardinality.

Dataset Composition

ROSE v0.1 comprises 1,512 scenes (3,024 images) and 7,560 task instances across five fine-grained visual sources to ensure robustness against specific semantic categories:

• CHINESEGLYPH: Visually confusable characters in a verified font.
• EMOJISTYLE: Same emoji identity, different rendering providers.
• EMOJICONTENT: Different emoji identities, shared rendering style.
• PIXELEDIT: Localized edits on the same pixel-art asset.
• PIXELCONTENT: Distinct but visually related pixel-art assets.

The dataset is split at the scene level to prevent data leakage between training/validation and test sets.

Key Contributions

1. Formulation of Reference-Conditioned Visual Action: A controlled setting that isolates the perception-to-action interface by fixing visual evidence while varying context and output requirements.
2. The ROSE Benchmark: A diagnostic dataset of 1,512 scenes and 7,560 instances across five visual sources and five task templates, including strict formal-output protocols to separate grammar compliance from task success.
3. Diagnostic Evaluation Framework: A suite of analyses including "bridge tasks" (Global-Click and Matched Local Count-to-Click) that decompose performance drops into specific failure modes: coordinate grounding, context-conditioned selection, and formal execution.

Experimental Results

The authors evaluated nine recent MLLMs (including GPT-5.5, Gemini-3.1-Pro, Qwen3.6-Plus, and Claude models) against a human reference.

Performance Gap

• Human Performance: Achieved 98.8% average PASS.
• Model Performance: Ranged from 14.3% to 92.2%.
• The Perception-to-Action Gap: A significant drop in performance occurs when moving from counting tasks to region-conditioned action. For example, Qwen3.6-Plus dropped from 80.3% on counting tasks to 37.7% on region-conditioned actions (a 42.6 percentage point drop). GLM-5V-Turbo dropped from 60.5% to 16.0%. Even high-performing GPT-5.5 saw a drop from 93.8% to 84.3%.

Decomposition of the Gap

• Coordinate Grounding vs. Context Selection: Bridge tasks revealed that while coordinate grounding (Global-Click) explains part of the loss, the majority of the performance drop occurs when adding region constraints. For Qwen3.6-Plus, the drop from Global Counting to Global Clicking was 13.2 points, but the additional drop to Visual-Region Clicking was 29.4 points.
• Consistency Failure: Even when models correctly counted the global exceptions, they frequently failed to filter these exceptions correctly under new region constraints. For Qwen3.6-Plus, conditioned on correct global counting, the accuracy for region-conditioned clicking remained low (38.0%).
• Failure Modes:
  • Invalid Output: Some models (e.g., Claude) failed primarily due to protocol violations (invalid grammar).
  • Region Violations: Stronger models (e.g., Qwen, GLM) often produced valid grammar but clicked cells outside the required region or failed to abstain when no targets existed.
  • Count Anchoring: Models often retained the global count or full set of coordinates even when the task required a subset or exclusion, failing to update the target set based on the new context.

Significance and Claims

The paper claims that ROSE reveals a distinct, model-dependent bottleneck in turning shared visual evidence into context-specific actions. The primary finding is that many models can "see enough to count" but not "enough to act."

• Beyond Perception: The results suggest that reliable multimodal action requires not just recovering visual evidence, but recomputing the context-specific target set and executing it exactly.
• Diagnostic Utility: ROSE provides a controlled testbed to distinguish between failures in perception, region binding, coordinate grounding, and action execution.
• Modest Scope: The authors explicitly state that ROSE is a controlled diagnostic benchmark, not a simulation of open-world interaction. It uses grid-structured scenes and symbolic actions to make perception and execution measurable, acknowledging that it does not capture the full complexity of natural images, free-form language, or long-horizon planning.

The paper concludes that while current MLLMs show remarkable progress, the transition from recognizing visual patterns to performing precise, context-dependent actions remains a significant challenge that requires further investigation into how models maintain consistency across changing task contexts.