Technical Summary: VLATIM Benchmark for Vision-Language Models in Point-and-Click Puzzle Games

1. Problem Statement

While Vision-Language-Action Models (VLAs) and Vision-Language Models (VLMs) have shown promise in integrating visual perception, language understanding, and action generation, existing benchmarks often fail to capture the specific challenges of continuous action spaces combined with complex physical reasoning. Current evaluations frequently rely on either overly simple discrete inputs (e.g., grid puzzles) or excessively complex open-world environments (e.g., Minecraft, Zelda) where feedback loops are either too restrictive or too noisy.

There is a critical gap in evaluating whether VLMs can perform human-like logical problem-solving in environments that require:

1. High-level reasoning: Understanding cause-and-effect relationships and physical mechanisms (e.g., chain reactions).
2. Precise execution: Translating that reasoning into continuous, precise mouse interactions (clicking, dragging, hovering) within a graphical interface.

The authors posit that while models may possess reasoning capabilities, the translation of these capabilities into precise physical actions within a game environment remains an unproven hypothesis.

2. Methodology: The VLATIM Benchmark

To address this gap, the authors introduce VLATIM (Vision-Language Against The Incredible Machine), a benchmark based on the classic physics puzzle game The Incredible Machine 2 (TIM). TIM is selected for its reliance on Rube Goldberg-style contraptions, requiring players to solve simple tasks through complex chain reactions involving gravity, friction, and mechanical interactions.

2.1 Benchmark Structure

The benchmark is divided into five progressive parts, ranging from zero-shot perception to full puzzle solving:

• Part 1: Visual Grounding: Evaluates object classification and localization (bounding boxes) using single-choice Q&A and coordinate outputs. Metrics include Intersection over Union (IoU) and Euclidean Distance.
• Part 2: Domain Understanding: Tests knowledge of object states (e.g., a cat running from a mouse) and physical properties (e.g., friction, mass). This includes text-based identification with and without explicit property descriptions.
• Part 3: Event Reasoning: Assesses the ability to reason about cause, effect, and outcomes in chain reactions. Tasks are split into textual (describing the event) and visual (marking the region of effect) components.
• Part 4: Manipulation: Requires the model to interact with the game interface via a vision-action loop. The action space includes: click, hover, drag, wait, and finished. Tasks involve placing, moving, rotating, stretching, and removing objects.
• Part 5: Full Puzzle Solving: The model is given only the high-level goal ("Solve the puzzle") and must read the objective from the screen, plan the solution, and execute the sequence of actions to complete the puzzle.

2.2 Experimental Setup

• Models Evaluated: Five VLMs were tested:
  • UI-TARS 1.5 (7B): A GUI agent trained for computer use.
  • Qwen2.5 VL (7B): The base model for UI-TARS.
  • Qwen3 VL (235B): A visually grounded agent with computer-use capabilities.
  • Gemini 2.5 Flash: A closed-source model with strong world knowledge.
  • GPT-5 Mini: A closed-source model with strong world knowledge.
• Evaluation Framework: For Parts 4 and 5, a vision-action loop was implemented where the model receives the current screenshot and action history, generates a thought and action, executes the action via pyautogui, and receives a new screenshot. Success rates were determined via human evaluation for complex tasks and string matching for classification.

3. Key Results

3.1 Disparity Between Reasoning and Execution

The results reveal a fundamental disconnect between a model's ability to reason about a scenario and its ability to execute the necessary physical actions.

• Visual Grounding (Part 1): Proprietary models (Gemini, GPT) demonstrated superior object classification but failed significantly at precise localization. They often output coordinates with low resolution (rounded to 10s) or relative coordinates that did not map accurately to bounding boxes. Conversely, Qwen2.5 and Qwen3 showed better localization capabilities but struggled with classification.
• Domain & Event Reasoning (Parts 2 & 3): Larger models (Qwen3, Gemini, GPT) outperformed smaller ones in textual reasoning. However, providing more context (e.g., full parts descriptions) did not consistently improve performance and, in some visual tasks, degraded it.
• Manipulation (Part 4): Success rates were generally low.
  • Simple tasks like "Place" and "Remove" saw some success.
  • Complex spatial tasks like "Rotate" and "Stretch" were rarely solved successfully.
  • No model successfully completed the "Multi" task (combining multiple actions).
• Full Puzzle Solving (Part 5): 0% Success Rate. None of the five models, regardless of size or architecture, successfully solved a complete puzzle from start to finish.

3.2 Failure Archetypes

The authors identify two distinct failure modes:

1. "Blind Strategists" (Gemini, GPT): These models possess strong reasoning capabilities, understanding physics and formulating logical plans. However, they fail at visual grounding, generating imprecise coordinates that prevent successful execution.
2. "Myopic Operators" (UI-TARS, Qwen2.5): These models can execute precise clicks and navigate the GUI effectively but lack the high-level reasoning to form multi-step plans, often getting stuck in loops or failing to recognize when a task is complete.
3. Qwen3 VL: Sits in the middle, attempting to bridge the gap with improved reasoning and grounding, but still fails to execute plans reliably enough to solve full puzzles.

4. Key Contributions

• VLATIM Benchmark: A novel benchmark specifically designed to evaluate the gap between logical reasoning and continuous action execution in a physics-based puzzle environment.
• Comprehensive Evaluation: A structured five-part evaluation covering visual grounding, domain knowledge, event reasoning, manipulation, and full puzzle solving.
• Empirical Evidence of Limitations: The study provides definitive evidence that current state-of-the-art VLMs do not yet possess human-like problem-solving capabilities in these environments, primarily due to the inability to simultaneously maintain high-level planning and precise low-level control.

5. Significance and Conclusion

The paper concludes that while VLMs are advancing rapidly, they currently fall significantly short of human-level performance in point-and-click puzzle games. The primary limitation is not a lack of reasoning power in isolation, but the inability to integrate precise visual grounding with complex logical planning.

The authors emphasize that the "Blind Strategist" and "Myopic Operator" archetypes highlight a need for future architectures that can better unify these modalities. The benchmark serves as a rigorous testbed for future research, suggesting that simply scaling models or providing more context is insufficient without addressing the specific challenges of continuous action spaces and precise spatial reasoning.

The paper modestly notes that while zero-shot performance is poor, future work could explore in-context learning (few-shot examples) or fine-tuning specifically for the game mechanics, though these were outside the scope of the current study. Ultimately, the findings suggest that achieving human-like autonomy in interactive physical environments requires more than just current VLM capabilities.