🤖 AI

Can Coding Agents Solve Repository-Level Issues with Rendered Code? An Exploratory Study of Visual Representations

This study evaluates the use of rendered code images as a token-compression strategy for repository-level coding agents, finding that while it effectively reduces prompt costs and maintains repair accuracy, its benefits are conditional and limited by the underlying model's capabilities and the specific stages of the repair workflow.

Original authors: Weijie Liang, Yuanfeng Song, Xing Chen, Caleb Chen Cao, Sirui Han, Yike Guo

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Weijie Liang, Yuanfeng Song, Xing Chen, Caleb Chen Cao, Sirui Han, Yike Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers are like incredibly fast, super-smart librarians who can read millions of books in a second. These "AI agents" are being taught to fix broken code in massive software projects, acting like digital mechanics for the internet. But there's a catch: these librarians get overwhelmed if you hand them a whole library at once. They need to read the text line-by-line, and the more text they have to process, the more it costs in time and money. Recently, scientists discovered a trick: instead of giving the librarian a wall of text, you can take a photo of the code. It's like turning a long novel into a single, dense picture. This "visual compression" saves space, but does it actually help the librarian fix the book, or does it just make the picture too blurry to read?

This question sits at the heart of a new study by researchers from Hong Kong University of Science and Technology and ByteDance. They wanted to know if turning code into images is a magic shortcut for AI agents trying to repair software, or if it's just a neat trick that hits a wall when the work gets real. They didn't just ask the AI to "look" at a picture; they put the AI in a realistic scenario where it had to hunt through a messy digital warehouse, find the broken part, fix it, and prove it worked. Their findings suggest that while taking a photo of the code saves a lot of money, it doesn't make the AI any smarter, and sometimes, if you squeeze the photo too tight, the AI gets confused.

The Experiment: From Text to Pixels

To test this, the researchers set up a digital obstacle course using a famous benchmark called SWE-bench Verified. Think of this as a giant, real-world video game level where the AI has to fix actual bugs in real software projects. The AI agent had to do three main things:

Search: Find the right file in a massive codebase (like finding a specific needle in a haystack).
Edit: Change the code to fix the bug.
Verify: Run tests to make sure the fix actually worked and didn't break anything else.

The researchers compared two ways of giving the AI the "clues" it needed:

The Text Way: The AI reads the code line-by-line, just like a human reading a book.
The Visual Way: The code is turned into an image (a rendered screenshot), and the AI looks at the picture to understand the code.

They tested this with different levels of "compression." Imagine taking a photo of a page of text. You can make the photo smaller (compressing it) to save space. The researchers tried making the photos smaller and smaller (compression ratios of 1, 3, 5, and 7) to see how much space they could save before the AI stopped understanding the code.

The Results: A Mixed Bag of Savings and Struggles

The study revealed a fascinating, slightly complicated picture. Here is what they found:

1. The Money Saver (But Not a Magic Wand)
Turning code into images definitely saves money. The researchers found that using visual representations consistently reduced the number of "tokens" (the units of data the AI processes) the AI had to read. In some cases, this saved up to 2.8 times the amount of data compared to reading raw text. However, this saving wasn't a straight line. You might think that if you compress the image by 7 times, you save 7 times the money. But that's not how it works. The savings hit a "floor" very quickly. For small chunks of code, the image size barely shrinks no matter how much you try to compress it, because the image still needs to be big enough to be readable. It's like trying to shrink a postage stamp; eventually, you can't make it any smaller without making the stamp unreadable.

2. The Accuracy Trap
The most important finding was that saving money didn't make the AI smarter. The AI's ability to actually fix the bugs (its accuracy) stayed roughly the same whether it read text or looked at pictures.

When the AI had to do the whole job from scratch (searching, reading, fixing), the visual method worked just as well as the text method, but cheaper.
However, if the researchers forced the AI to compress the images too much (aggressive compression), the AI started making more mistakes. The pictures became too blurry or crowded, and the AI couldn't see the details it needed to fix the code.

3. The "Search" vs. "Fix" Problem
The researchers broke the AI's job into two distinct phases to see exactly where the visual trick helped:

The Locator (The Searcher): This part of the AI looks for where the bug is. Here, visual images were very helpful. Since the AI had to read a lot of raw code to find the needle in the haystack, turning that code into a compact image saved a huge amount of data.
The Editor (The Fixer): Once the bug is found, the AI has to actually change the code and run tests. Here, the visual trick didn't help much. The researchers found that even if the AI knew exactly where the bug was, the hardest part was the "trial and error" of fixing it. The AI had to edit, test, fail, edit again, and test again. This process is full of back-and-forth conversation and testing, which doesn't benefit much from compressing the initial code image. In fact, for these "fixing" steps, the cost was dominated by the testing and editing, not the reading.

The Verdict: A Useful Tool, But Not a Cure-All

So, can coding agents solve repository-level issues with rendered code? Yes, but with conditions.

The study suggests that turning code into images is a viable strategy to cut costs, especially when the AI is spending a lot of time just reading and searching through massive amounts of text. It's like giving the librarian a photo of the book's index instead of the whole book to find a page number. It's faster and cheaper.

However, it is not a magic solution that makes the AI infinitely better.

It doesn't fix the AI's underlying intelligence; if the AI is bad at fixing bugs, it will still be bad at fixing bugs even if it sees a picture.
It has a limit. If you try to squeeze the image too small to save even more money, the AI gets confused, and the accuracy drops.
It helps most in the "search" phase. Once the AI knows where the problem is, the real work of fixing it (editing and testing) becomes the bottleneck, and visual compression doesn't help much there.

In short, the researchers conclude that visual code is a great "conditional" tool. It's a smart way to save money on the boring, heavy lifting of reading code, but it doesn't replace the need for a smart, careful editor to actually do the repair work. The future of AI coding isn't just about making the input smaller; it's about knowing when to use a picture and when to stick to the text.

Technical Summary: Can Coding Agents Solve Repository-Level Issues with Rendered Code?

Problem Statement

Recent advancements in long-context models and agentic systems have highlighted the computational bottleneck of processing large amounts of textual information. While prior research has explored converting textual information into visual forms (e.g., rendering code as images) to compress context for static code understanding, a critical gap remains: it is unclear whether these visual representations can serve as effective operational context for agentic coding.

Unlike static understanding tasks where a model passively "reads" code, repository-level repair agents must actively navigate file systems, localize relevant regions, edit source files, execute tests, and iteratively refine hypotheses. The core research question is whether rendering code as images can reduce token costs and maintain accuracy within these complex, interactive, repository-level workflows, and where the benefits of such compression are actually realized.

Methodology

The authors conduct a systematic exploratory study using the SWE-bench Verified benchmark, evaluating rendered code across three distinct agent configurations to isolate the effects of visual modality from other variables.

1. Agent Configurations

To address the "attribution challenge" (distinguishing between modality effects and trajectory variance), the study introduces a controlled progression of agent designs:

Mini SWE-agent (Baseline): A standard, single-stage bash-first agent that interleaves repository exploration, code reading, editing, and testing. This represents the unguided, native workflow.
Pre-Localized Bash Agent (PLB): This agent performs an offline, structured localization pass (using a static repository index and graph) before entering the repair loop. This decouples unguided exploration from the repair stage, allowing the study to measure visual compression benefits when the agent starts with a known context.
Locator–Editor Tool Agent (LET): This agent explicitly separates the workflow into two stages:
- Locator: Uses structured tools (Localize, Package, view_code) to identify relevant files and symbols and commit a structured plan.
- Editor: Receives the plan and focuses on trial-and-error editing, testing, and patch refinement.
- Note: In the visual LET experiments, rendering is applied only to the Locator phase to test if visual compression retains value once structured information has already reduced the need for raw source reading.

2. Experimental Setup

Backbones: Experiments utilize two multimodal models: Qwen3.5 and Kimi-K2.5.
Conditions: Agents operate under one text condition and four visual conditions with nominal compression ratios ( $r \in \{1, 3, 5, 7\}$ ).
Rendering Pipeline: A custom renderer maps code windows to images, searching over discrete layout configurations (resolution, font size, pagination) to meet a target token budget while preserving readability.
Metrics: Evaluation focuses on end-to-end repair accuracy (pass@1) and prompt-token consumption.

Key Findings

1. Token Cost Reduction vs. Linear Scaling

Rendered code consistently reduces prompt-token costs compared to raw text. However, the savings do not scale linearly with the nominal compression ratio ( $r$ ).

Discrete Cost Floor: Small code windows quickly hit a visual cost floor because they collapse to the same rendering configuration regardless of $r$ .
Diminishing Returns: For larger windows, higher compression ratios yield savings, but with diminishing marginal returns.
Readability Trade-off: Aggressive compression (high $r$ ) can degrade readability, leading to unstable performance or increased token usage due to repeated reading attempts.

2. Accuracy Preservation and Stability

General Stability: Across both PLB and LET configurations, rendered code largely preserves end-to-end repair accuracy compared to the text baseline.
Instability at High Compression: In the LET setting, aggressive compression (e.g., $r=7$ ) caused a noticeable drop in accuracy (down to 60.0% on Kimi-K2.5), whereas PLB remained more stable. This suggests that when the agent relies heavily on visual context for the entire repair loop, readability issues become more critical.

3. Mechanisms of Benefit

The study identifies three mechanisms shaping the utility of rendered code:

Reshaping Behavior: Visual compression reshapes repair behavior by compressing unguided exploration into the early trajectory, shifting later actions toward focused test–edit repair.
Context Dependency: Visual compression is most beneficial when raw source reading is a major bottleneck. In the PLB setting (where exploration is structured but reading is still raw), visual compression achieved the highest token reductions (up to 2.81x).
Marginal Utility Limits: Once structured repository information (as in the LET Locator phase) reduces the demand for raw source reading, the marginal benefit of visual compression shrinks. In LET, the dominant cost shifts to the Editor phase (editing, testing, and patch iteration), where visual compression offers limited leverage.

4. The Role of Trial-and-Error

Analysis of the LET decomposition reveals that trial-and-error matters more than repository understanding alone.

Even with precise localization, the Editor phase consumes the majority of tokens (up to 83.7% in non-resolved tasks).
Better localization improves the initial conditions for the editor but does not guarantee a successful patch. The final resolution is often dominated by the cost of editing, testing, and refining patches, a process where visual compression has limited impact.

Significance and Claims

The paper positions rendered code as a viable but conditional compression mechanism for realistic coding agents.

Conditional Viability: The benefit of visual representation is not universal; it is highly dependent on the workflow stage. It is most effective when the agent is actively reading raw source code to localize issues.
Workflow Integration: The central challenge for repository-level repair is not merely how to compress code, but where to place that compression within the broader edit workflow. Once the workflow is structured to minimize raw reading (e.g., via offline localization), the remaining costs are driven by iterative patch refinement, a domain where visual compression offers diminishing returns.
Practical Implications: For production-grade agentic systems, visual compression should be strategically deployed during the exploration and localization phases rather than as a blanket replacement for all textual context, especially given the non-smooth cost surface and readability risks of aggressive compression.

The authors conclude that while visual modality offers a promising route for token efficiency, its success in agentic coding is bounded by the specific mechanics of the repair loop and the inherent trade-offs between compression density and code readability.