💻 computer science

GALA: Multimodal Graph Alignment for Bug Localization in Automated Program Repair

GALA is a novel framework for multimodal Automated Program Repair that leverages hierarchical graph alignment to bridge the gap between GUI screenshots and source code, achieving state-of-the-art bug localization and repair performance by replacing implicit semantic guessing with explicit structural reasoning.

Original authors: Zhuoyao Liu, Zhengran Zeng, Shu-Dong Huang, Yang Liu, Shikun Zhang, Wei Ye

Published 2026-04-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhuoyao Liu, Zhengran Zeng, Shu-Dong Huang, Yang Liu, Shikun Zhang, Wei Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to fix a broken vending machine.

The Old Way (Previous Methods):
You are handed a photo of the machine where a candy bar is stuck, and a note that says, "The chocolate bar won't drop."
The old detectives (AI models) would take that photo, describe it in words like "There is a chocolate bar and a button," and then search through a massive library of blueprints (the code) looking for the words "chocolate" or "button."
The Problem: They often get confused. They might find a blueprint for a different machine that also has a "button" and a "chocolate bar," but it's the wrong machine entirely. They miss the crucial detail: how the button is physically connected to the mechanism that drops the candy. They are guessing based on keywords, not understanding the structure.

The New Way (GALA):
The authors of this paper, GALA, propose a smarter detective. Instead of just reading the photo, GALA turns the photo into a map and the code into a roadmap, then matches them up perfectly.

Here is how GALA works, step-by-step, using our vending machine analogy:

1. Drawing the "Bug Map" (Image Graph Construction)

Instead of just describing the photo, GALA looks at the screenshot of the broken app (or vending machine) and draws a diagram.

It identifies the specific parts: "This is the 'Buy' button," "This is the 'Error' message," and "This is the 'Price' display."
Crucially, it draws lines connecting them to show relationships: "The 'Buy' button triggers the 'Error' message," or "The 'Price' display is next to the button."
The Magic: It ignores the background noise (like the wall behind the machine) and only focuses on the parts involved in the bug. It creates a structured "skeleton" of the visual problem.

2. Finding the Right Blueprint (File-Level Alignment)

Now, GALA has this "Bug Map." It needs to find the right blueprint in the giant library of code.

Instead of searching for the word "button," GALA looks at the structure of the code library. It sees how files are connected (e.g., main.js imports button.js).
It matches the "Bug Map" to the "Code Library Map." It asks: "Which group of files has a structure that looks like our visual map?"
Result: It narrows down the search from the whole library to just 5 or 6 specific files that are likely the culprits.

3. Pinpointing the Exact Screw (Function-Level Alignment)

Okay, we have the 5 files. But which line of code is broken?

GALA zooms in further. It creates a mini-map of the functions inside those files (e.g., checkPrice(), displayError()).
It matches the visual relationships to the code relationships.
- Visual: "The button triggers the error."
- Code: "The onClick function calls the showError function."
The Match: It finds the exact function that corresponds to the visual trigger. It's like finding the exact screw on the vending machine that is loose, rather than just saying "the button assembly is broken."

4. Fixing the Machine (Patch Generation)

Now that GALA knows exactly which screw is loose and how it connects to the rest of the machine, it writes a precise repair instruction (a "patch").

Because it understands the structure and not just the words, the fix is accurate. It doesn't accidentally break the coin slot while fixing the button.

Why is this a big deal?

Old AI: "I see a button. I see a file named 'button.js'. Let's fix that." (Often wrong).
GALA: "I see a button that triggers an error. In the code, the click function calls the error function. The connection between them is broken. Let's fix that specific link." (Usually right).

The Result

The paper tested this on a huge benchmark of real-world software bugs (SWE-bench Multimodal). GALA solved more bugs than any other method, proving that understanding the structure and relationships (the "skeleton") is much better than just matching keywords.

In short: GALA stops the AI from guessing based on a description and starts it by building a structural map of the problem, then matching that map to the code's blueprint to find the exact fix. It's the difference between a detective guessing who the criminal is based on a description, and one who has a fingerprint match.

1. Problem Statement

Automated Program Repair (APR) systems based on Large Language Models (LLMs) have shown success in textual benchmarks but struggle significantly in multimodal scenarios where bug reports include GUI screenshots.

Current Limitations: Existing multimodal APR methods typically convert images into plain text descriptions using Vision-Language Models (VLMs). This approach suffers from two critical flaws:
1. Loss of Structural Relationships: Converting images to text discards complex spatial and structural relationships between UI elements (e.g., "overlap," "containment," "adjacency"), which are crucial for understanding layout bugs.
2. Visual-Code Disconnect: Localization relies on implicit semantic keyword matching rather than explicit structural linking. This leads to retrieving files with similar semantics but unrelated to the actual visual bug, causing imprecise localization.
Goal: To bridge the gap between visual observations and code components by shifting from implicit semantic guessing to explicit structural reasoning.

2. Methodology: The GALA Framework

GALA (Graph Alignment for Localization in APR) proposes a four-stage pipeline that treats multimodal localization as a hierarchical cross-modal graph alignment problem.

Stage 1: Image UI Graph Construction

Instead of generating a caption, GALA constructs a structured Image UI Graph ( $G_v = (V_v, E_v)$ ) to preserve visual topology.

Type Identification: A VLM classifies the image (e.g., UI page, chart, code screenshot) to guide the extraction strategy.
Root Object Selection: The model identifies "root nodes" directly related to the bug (e.g., a specific button or error message) based on visual evidence and the issue description.
Supporting Node Expansion: The model expands the graph to include only necessary one-hop context (neighboring elements, containers) required to explain the root objects, avoiding irrelevant global scene details.
Relation Construction: Directed edges are added to represent spatial and functional relationships (e.g., contain, triggers, overlaps) with explicit justifications.

Stage 2: File-Level Alignment

This stage grounds the visual semantics into the repository structure to identify a compact set of seed files.

Repository Grounding: The code repository is parsed into a hierarchical directory tree, filtering out irrelevant paths (e.g., node_modules).
Candidate Retrieval: The model uses the issue description, the repo tree, and a summarized version of the Image UI Graph to retrieve a list of candidate files.
Structure-Aware Refinement: A File Graph is constructed where nodes are files and edges represent inter-file dependencies (imports). The model performs graph alignment to select seed files that are not only semantically relevant but also structurally consistent with the visual issue.

Stage 3: Function-Level Alignment

Within the selected seed files, GALA performs fine-grained alignment to pinpoint exact edit targets.

Function Graph Construction: A closed-domain Function Graph ( $G_c = (V_c, E_c)$ ) is built from the seed files. Nodes represent functions/methods, and edges represent UI-relevant interactions (e.g., renders, writes_state, calls).
Cross-Modal Alignment: The model aligns nodes from the Image UI Graph with nodes in the Function Graph. It verifies that semantic matches are supported by consistent relational evidence (e.g., if the image shows a button triggering an error, the code graph must show a function call leading to state change).
Target Identification: The output is a small set of aligned subgraphs identifying specific functions as primary, secondary, or contextual edit targets.

Stage 4: Graph-Guided Patch Generation

The repair agent generates code patches constrained by the aligned graph structure.

Localized Repair Space: The agent focuses on the identified seed files and edit targets, expanding only if dependency evidence suggests adjacent components are involved.
Target Prioritization: Edits are prioritized based on the alignment roles (primary targets first).
Output: The system generates a minimal, executable patch (git diff) that resolves the visual symptom.

3. Key Contributions

Reformulation of Localization: Shifts multimodal bug localization from text-based retrieval to a hierarchical cross-modal graph alignment problem.
Structured Representations: Proposes an Image UI Graph to capture visual semantics and multi-level Code Graphs (file and function levels) to model code dependencies explicitly.
Hierarchical Alignment Mechanism: Designs a mechanism that enforces both semantic and relational consistency across modalities, bridging visual elements to code from the file level down to the function level.
State-of-the-Art Performance: Demonstrates that structure-aware alignment significantly outperforms existing methods, validating the approach on the SWE-bench Multimodal benchmark.

4. Experimental Results

The framework was evaluated on the SWE-bench Multimodal benchmark (619 tasks from 17 JavaScript repositories).

Main Results:
- GALA achieved a 35.40% resolution rate (Pass@1) using the Qwen3.5-122B-A10B model.
- This outperformed the previous best methods: GUIRepair (34.82%) and OpenHands-Versa (34.43%).
- Crucially, under the same base model (122B), GALA surpassed SVRepair (33.66%), proving the improvement comes from the alignment strategy, not just model scaling.
Localization Accuracy:
- File-Level Recall: 29.22% (vs. 28.71% for SVRepair).
- Function-Level Recall: 17.14% (vs. 16.25% for SVRepair).
- GALA showed even greater relative gains on smaller models (35B), suggesting the structural guidance compensates for limited model capacity.
Ablation Studies:
- Removing the Cross-Modal Alignment (aligning Image Graph to Code Graph) dropped performance significantly (from 35.40% to 34.43%), confirming that explicit structural alignment is the key driver of success.
- Adding Function-Level Graphs provided further gains over File-Level graphs alone, highlighting the importance of fine-grained reasoning.

5. Significance

Paradigm Shift: GALA moves the field away from treating images as mere text supplements. It establishes that explicit structural reasoning is necessary for multimodal software engineering tasks.
Robustness: The method provides robust guidance across different model scales, making it effective even with smaller, more efficient LLMs.
Future Direction: The work highlights the potential of using structured intermediate representations (graphs) to scaffold LLM reasoning, potentially paving the way for future systems that internalize these structures for direct end-to-end reasoning.

In summary, GALA successfully addresses the disconnect between visual bug reports and code by constructing and aligning structured graphs, resulting in more precise bug localization and higher repair success rates in multimodal settings.