Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing
The paper introduces GUIRepair, a cross-modal reasoning framework that enhances multimodal automated program repair by integrating Image2Code and Code2Image components to translate visual GUI symptoms into executable context and validate fixes through visual feedback, achieving state-of-the-art performance on the SWE-bench M benchmark.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a master mechanic trying to fix a very specific, tricky car problem. Usually, the car owner calls you and says, "My engine is making a weird noise," and you fix it based on that description. This is how most current AI "program repair" tools work: they read text descriptions of bugs and look at the code to fix them.
But what if the problem is visual? What if the car owner says, "The dashboard light is blinking the wrong color," and sends you a photo? Most AI mechanics today get confused. They can read the text, but they can't really "see" the photo or understand how that picture connects to the engine parts. They struggle to figure out which bolt to tighten just by looking at a snapshot.
This paper introduces a new AI mechanic called GUIRepair. It's designed specifically to handle these "visual" software bugs, like a broken button on a website or a glitchy map on an app.
Here is how GUIRepair works, broken down into two main superpowers:
1. The "Photo-to-Blueprint" Translator (Image2Code)
When a human developer sees a bug in a photo, they don't just stare at the pixels; they think, "Ah, that's the Calendar component acting up because of how the Dialog box is set up." They translate the picture back into the code logic in their head.
AI usually can't do this well. GUIRepair fixes this by acting like a detective who builds a miniature model of the problem.
- The Process: It looks at the bug report (the photo and the text) and searches the project's instruction manuals (documentation) to understand the rules.
- The Magic: It then writes a tiny, temporary piece of code that recreates the exact scene from the photo. It's like the AI building a "test drive" version of the car just to see the blinking light for itself.
- Why it helps: Now, instead of guessing what the photo means, the AI has a working model. It can see, "Oh, I see exactly which line of code is causing that light to blink." This helps it find the right place to fix the bug.
2. The "Fix-and-Check" Mirror (Code2Image)
Once the AI thinks it has a fix, it needs to know if it actually worked. In normal text-based coding, you run a test that says "Pass" or "Fail." But for visual bugs, a text test isn't enough. You need to see if the light is now the right color.
GUIRepair has a second superpower to handle this:
- The Process: It takes the fix it just wrote and applies it to the "miniature model" it built earlier.
- The Magic: It runs the code and takes a new screenshot of the result. It then compares this new picture with the original "broken" picture.
- Why it helps: The AI looks at the two images side-by-side and says, "Okay, in the first picture, the calendar was floating in the wrong spot. In this new picture, it's in the right spot. The fix worked!" This gives the AI a way to "see" if its repair is successful, rather than just guessing.
The Results: A Better Mechanic
The researchers tested this new mechanic on a huge list of real-world software bugs (called SWE-bench M) that involve visual problems.
- The Competition: They compared GUIRepair against the best existing AI systems (both free and paid commercial ones).
- The Score:
- Using a standard powerful AI model, GUIRepair fixed 157 problems, beating the next best open-source tool by a significant margin.
- When they used an even smarter "reasoning" AI model (called o4-mini), GUIRepair fixed 175 problems, beating the top commercial system by 22 fixes.
The Bottom Line
The paper claims that by teaching the AI to translate pictures into code (to understand the problem) and translate code back into pictures (to check the solution), it can solve visual software bugs much better than current methods. It's like giving the mechanic a pair of glasses so they can finally "see" the problem they are trying to fix.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.