Technical Summary: Self-Evolving Code-with-Image Reasoning

1. Problem Definition

Current multimodal reasoning paradigms, specifically "thinking-with-images," rely on models invoking a fixed set of tools (e.g., cropping, zooming, brightness adjustment) to gather visual evidence, after which reasoning resumes in natural language. The paper identifies a fundamental ceiling in this approach: while tools can expose evidence, they cannot execute multi-step visual algorithms over pixel data. Many visual questions—such as image forensics, precise geometric transformations, or signal measurements—require exact computations over arrays that are invisible to human inspection and cannot be solved by a chain of linguistic tokens alone.

In these scenarios, a model may correctly identify the necessary algorithm in text but fail to execute it because language cannot manipulate data structures like arrays or masks. The bottleneck shifts from perceiving the image to computing the answer. Existing benchmarks often test tool usage or retrieval rather than the ability to author and execute a novel visual algorithm from scratch.

2. Methodology

The Code-with-Image Regime

The authors propose a new reasoning paradigm called Code-with-Image. In this regime, the environment provides only a general-purpose Python interpreter. The model is not given a pre-defined tool vocabulary. Instead, it must:

1. Receive the image as raw data.
2. Author a Python program to implement a visual algorithm.
3. Execute the program, where the program's output (data arrays, computed values) constitutes the reasoning process.
4. Transcribe the final program output as the answer.

Here, the program itself is the reasoning; debugging the code is equivalent to debugging the reasoning.

Benchmark: CwI-Bench

To isolate and evaluate this regime, the authors introduce CwI-Bench, a benchmark consisting of 30 task families across four semantic categories: Transformation, Perception, Localization, and Color.

• Constructive Ground Truth: Instances are generated by algorithms that construct images from known targets (e.g., rotating a grid by a specific angle), ensuring the labels are correct by construction rather than human annotation.
• Beyond Inspection: Targets are precise quantities (e.g., exact pixel coordinates, rotation angles) that cannot be determined by visual inspection, even with zooming.
• Disjoint Splits: The benchmark features disjoint training, validation, and test splits with zero source-image intersection to measure genuine procedure learning rather than memorization.

Self-Reflection over Executable Reasoning

To improve performance without weight updates, the authors propose a training-free, two-mode reflection loop that evolves a library of "skills" (plain text descriptions of procedures).

1. Observational Reflection: The model reads multimodal trajectories of its own failures and successes. It proposes bounded edits to the skill library (adding, replacing, or deleting procedures) based on textual analysis of the error.
2. Executable Reflection: When observational reflection stalls, the model enters a sandboxed debugging session. It re-executes failed programs, renders intermediate states (arrays, masks), and tests candidate repairs against ground truth. This mode addresses "silent faults" (e.g., off-by-one errors, unsigned arithmetic wraps) that do not appear in the printed text output but are visible in the data.

The loop iterates, selecting skills that improve accuracy on a held-out validation set. The final skills are injected as text into the system prompt of the frozen model.

3. Key Contributions

• CwI-Bench: A 30-task benchmark designed specifically for "program-native" reasoning, where the answer is the output of an authored program rather than a linguistic claim. It demonstrates that even advanced models (e.g., GPT-5.6-luna) struggle below 30% accuracy without code, highlighting the gap between describing and executing algorithms.
• Self-Reflection over Executable Reasoning: A novel, training-free skill-evolution loop. It combines observational analysis with executable debugging to repair silent implementation faults. This allows models to accumulate portable, plain-text skills that improve performance from 32.6% to 55.9% on a 27B model and from 43.0% to 66.6% on a frontier model.
• Transferable Skills: The evolved skills are model-agnostic plain text. The paper demonstrates that skills evolved by a larger model (27B) can significantly boost the performance of a smaller model (9B), and skills can transfer across different model families (e.g., from Qwen to Gemma), provided the recipient model has the coding robustness to execute the procedures.

4. Experimental Results

• Baseline Performance: On CwI-Bench, tool-free Chain-of-Thought (CoT) reasoning yields very low accuracy (7–13% for open models, ~25% for frontier models in thinking mode).
• Code-with-Image Impact: Providing a bare Python interpreter significantly raises the floor. For example, GPT-5.6-luna improves from 25.0% (Thinking CoT) to 43.0% (Code-with-Image). The 27B Qwen model improves from 14.4% to 32.6%.
• Skill Evolution: Applying the self-reflection loop further boosts performance. The 27B model reaches 55.9%, and the frontier model reaches 66.6%.
• Ablation: Removing "Executable Reflection" (relying only on observational reading) results in a significant drop in performance (a 6.0 percentage point drop in mean uplift for the 27B model), particularly on Transformation tasks where silent implementation faults are common.
• Transfer: Transferring skills from a 27B model to a 9B model raises the 9B's accuracy from 11.1% to 34.5%, outperforming the 9B's own self-evolved skills (30.6%).

5. Significance and Claims

The paper claims that the primary bottleneck in complex visual reasoning is not the lack of perception tools, but the inability to execute multi-step algorithms over data. By shifting the reasoning medium from language to code, the "program" becomes the reasoning process.

The significance lies in demonstrating that:

1. Reasoning can be decoupled from weights: Models can learn complex visual procedures through a training-free loop that evolves text-based skills, making the reasoning process inspectable and portable.
2. Debugging code is debugging reasoning: Silent implementation errors are the primary failure mode in visual computation, and these can only be resolved by re-executing the computation and inspecting intermediate data states, not by re-reading the text.
3. Scalability of skills: Visual reasoning procedures are transferable assets. A model does not need to re-learn an algorithm; it only needs to be provided with the correct executable procedure, which can be generated by a stronger model or evolved through reflection.

The authors conclude that when code carries the reasoning, improving a model becomes a matter of debugging its programs, offering a practical route to accumulating and sharing visual-reasoning capabilities across models without parameter updates.