💻 computer science

Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents

This paper introduces Rust-SWE-bench, a large-scale benchmark for evaluating LLM-based agents on repository-level Rust issue resolution, and proposes RUSTFORGER, a novel agent that leverages dynamic tracing and automated test setup to significantly outperform existing baselines by overcoming challenges in code structure comprehension and strict type semantics.

Original authors: Jiahong Xiang, Wenxiao He, Xihua Wang, Hongliang Tian, Yuqun Zhang

Published 2026-02-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiahong Xiang, Wenxiao He, Xihua Wang, Hongliang Tian, Yuqun Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🦀 The Rust Problem: A High-Performance Language with a Steep Cliff

Imagine Rust as a super-safe, high-speed race car. It's built so well that it almost never crashes (no memory leaks or data races), making it perfect for building critical things like operating systems and web browsers.

However, driving this car is incredibly hard. The rules are strict, the controls are complex, and if you make even a tiny mistake, the car won't start at all. Because it's so difficult, many developers struggle to fix bugs or add new features. They need a mechanic who can understand the entire engine and fix it without breaking anything.

🤖 The Goal: Can AI Be That Mechanic?

Recently, we've seen Artificial Intelligence (AI) get really good at writing code. Scientists wanted to know: Can an AI agent fix real-world Rust bugs on its own?

To find out, they needed a "driving test" for the AI. But there was a problem: there was no big, realistic test for Rust. Existing tests were like driving a toy car in a parking lot, not racing on a real track.

🧪 Step 1: Building the "Rust-SWE-bench" (The Ultimate Driving Test)

The researchers built a new, massive test called Rust-SWE-bench.

The Track: They gathered 500 real-world problems from 34 popular, complex Rust projects (like the tools used by major tech companies).
The Challenge: The AI is given a description of a bug and the entire codebase. It must write a single "patch" (a fix) that makes the code work again and pass all the safety checks.
The Result: They created the first-ever "Olympics" for Rust-fixing AI.

🔍 Step 2: The Race (What Happened?)

They tested four different AI "mechanics" (agents) using four different "brains" (Large Language Models). Here is what they found:

The "ReAct" Style Wins: The best AI agents were those that used a "Think-Act-Observe" loop. Imagine a detective who thinks about a clue, goes to the scene to look, sees what happens, and then thinks again. This method worked best, solving about 21% of the problems.
The Bottleneck: The AI got stuck on two main things:
- The Map Problem: The AI couldn't understand how the whole library fit together. It was like trying to fix a specific wire in a house without knowing where the fuse box is.
- The Language Barrier: Rust has very strict grammar rules. The AI often wrote code that looked right but broke the rules, causing the "car" to refuse to start.
The Missing Link: The biggest failure point was reproducing the bug. Before you can fix a problem, you have to be able to make it happen again. The AI struggled to set up the environment to trigger the bug, so it couldn't even start fixing it. 44.5% of tasks failed just because the AI couldn't make the bug appear.

🛠️ Step 3: The Solution - "RustForger"

Inspired by these failures, the researchers built a new, super-powered mechanic called RustForger. Think of it as giving the AI a magic workshop and a super-spy camera.

The Magic Workshop (Isolated Environment): Instead of trying to fix the car in a messy, crowded garage, RustForger builds a clean, empty, isolated garage. It copies the car parts there, sets up the tools perfectly, and ensures nothing from the outside world interferes. This solves the "Map Problem."
The Super-Spy Camera (Dynamic Tracing): This is the secret sauce. RustForger uses a special "Trace" command. It's like putting a tiny camera on every moving part of the engine. When the engine runs, the camera records exactly what happens, step-by-step, in real-time.
- Why this matters: Instead of guessing why the engine stalled, the AI gets a video replay of the exact moment it failed. It can see, "Oh, this wire wasn't connected to that bolt."

🏆 The Results: A New Champion

When they put RustForger on the track:

Success Rate: It solved 28.6% of the problems (a huge jump from the previous best of 21.2%).
Unique Wins: It solved 46 problems that no other AI could solve, even with the most powerful brains.
Efficiency: It was also cheaper and faster because it didn't waste time guessing; it used the "camera" to see the truth immediately.

💡 The Big Takeaway

This paper teaches us two main lessons:

Context is King: To fix complex software, AI needs to understand the whole picture, not just the specific line of code.
Reproduction is Key: You can't fix what you can't see. Giving AI the ability to safely recreate and "watch" a bug in action is the most important step to solving it.

RustForger proves that by combining a clean workspace with a "live camera" view of the code running, we can teach AI to become a much better mechanic for even the trickiest languages.

1. Problem Statement

The Rust programming language offers high performance and memory safety but presents a steep learning curve due to its strict type system, ownership model, and complex trait semantics. While Large Language Model (LLM)-based agents have shown success in automating software engineering tasks for languages like Python, their application to Rust is hindered by two main factors:

Lack of Benchmarks: Existing benchmarks for Rust are either too small-scale, focused on granular tasks (e.g., function synthesis), or lack repository-level complexity.
Agent Limitations: Current agents struggle with Rust-specific challenges, particularly understanding repository-wide code structures and reproducing issues in Rust's strict compilation environment.

The core research question is: Can agentic approaches effectively automate the resolution of real-world, repository-level Rust issues, and if not, what are the bottlenecks and how can they be overcome?

2. Methodology

The authors propose a three-pronged methodology: constructing a new benchmark, conducting an extensive empirical study, and proposing a novel agent framework.

A. Benchmark Construction: Rust-SWE-bench

To fill the gap in evaluation data, the authors constructed Rust-SWE-bench, a large-scale benchmark containing 500 real-world, repository-level tasks from 34 popular open-source Rust repositories (e.g., tokio, bevy, clap, ripgrep).

Construction Process:
1. Selection: Sourced ~80,000 merged Pull Requests (PRs) from 87 high-star repositories.
2. Filtering: Selected PRs linked to resolved issues that included test modifications.
3. Validation: Ensured a "fail-to-pass" transition (tests fail before the patch, pass after). Compilation errors in Rust are treated as test failures.
4. Manual Verification: Inspected 1,040 candidates to remove solution leakage and ensure unambiguous problem descriptions, resulting in 500 high-quality tasks.
Characteristics: The benchmark features complex tasks requiring significant code changes (avg. 139.9 lines edited vs. 14.3 in Python benchmarks) and covers diverse categories like bug fixes, feature implementations, and API evolution.

B. Empirical Study

The authors evaluated four representative agents (SWE-agent, OpenHands+CodeAct, Agentless, AutoCodeRover) paired with four state-of-the-art LLMs (Claude-Sonnet-3.7, GPT-4o, OpenAI o4-mini, Qwen3) on Rust-SWE-bench.

Key Findings:
- Architecture: ReAct-style agents (iterative "thought-act-observe" loops) outperformed structured, single-pass workflows.
- Performance: The best baseline (OpenHands + Claude-Sonnet-3.7) resolved only 21.2% of tasks.
- Bottlenecks:
  1. Code Structure: Agents frequently failed to model repository-wide structures, leading to unresolved imports and path errors (e.g., E0433, E0432).
  2. Semantics: Agents struggled with Rust's strict type and trait constraints, causing compilation errors (e.g., E0599, E0277).
  3. Issue Reproduction: A critical bottleneck where 44.5% of tasks failed at the reproduction stage due to dependency misconfigurations or non-compiling test cases. Without successful reproduction, agents could not gather runtime data to fix the issue.

C. Proposed Solution: RustForger

Inspired by the finding that issue reproduction is critical, the authors introduced RustForger, a novel agentic framework designed to decouple issue reproduction from the complex build systems of the original project.

Core Components:
1. Automated Testing Workspace: Instead of running tests in the original project, RustForger creates an isolated, sandboxed Cargo workspace. It imports the target project as a local dependency, ensuring a clean environment for reproduction.
2. Trace Command (Cross-Project Dynamic Tracing): A unique mechanism leveraging Rust's metaprogramming (procedural macros) and AST modification.
  - The agent injects tracing macros into the target codebase's source files.
  - It executes tests from the isolated workspace, capturing runtime control-flow and data without being entangled in the original project's build system.
  - Data is serialized (handling Rust's non-serializable types via placeholders) and returned to the agent for root cause analysis.
Workflow: Setup Workspace $\rightarrow$ Reproduce Issue $\rightarrow$ Dynamic Tracing (if reproducible) $\rightarrow$ Patch Generation $\rightarrow$ Verification.

3. Key Results

Performance Improvement:
- RustForger + Claude-Sonnet-3.7 achieved a 28.6% resolution rate on Rust-SWE-bench.
- This represents a 34.9% relative improvement over the strongest baseline (OpenHands + Claude-Sonnet-3.7 at 21.2%).
- The improvement is statistically significant ( $p < 0.001$ via McNemar's test).
Unique Solvability: RustForger uniquely solved 46 tasks that no other agent (across all tested LLMs) could resolve.
Cost Efficiency: Despite higher resolution rates, RustForger incurred lower average API costs per task compared to baselines (e.g., 21.3% lower cost with Claude-Sonnet-3.7) due to reduced token usage from more efficient dynamic analysis.
Reproduction Success: RustForger improved the reproduction success rate to 67.3% (vs. 55.5% for OpenHands), directly addressing the primary bottleneck identified in the study.
Ablation Study: Removing the isolated workspace reduced performance by ~3 tasks; removing the Trace command reduced performance by ~5 tasks, confirming the value of both components.

4. Key Contributions

Rust-SWE-bench: The first large-scale, repository-level benchmark dedicated to real-world Rust software engineering issues, enabling systematic evaluation of LLM agents in the Rust ecosystem.
Comprehensive Study: A detailed analysis revealing that current agents fail primarily due to difficulties in modeling repository-wide code structures and reproducing issues in Rust's strict environment.
RustForger: A novel framework that integrates automated test environment setup with a cross-project dynamic tracing strategy powered by Rust metaprogramming. It effectively bypasses build complexities and provides agents with precise runtime insights.

5. Significance

This work is significant for several reasons:

Advancing Rust Automation: It demonstrates that with the right architectural interventions (specifically dynamic tracing and environment isolation), LLM agents can overcome the unique semantic and structural hurdles of Rust.
Benchmarking Standard: Rust-SWE-bench provides a rigorous, verified dataset that will likely become the standard for future research in Rust-specific automated program repair.
Methodological Insight: The paper highlights that for complex, compiled languages, dynamic analysis (via tracing) is often superior to purely static analysis for issue resolution, suggesting a new direction for agent design in systems programming.
Practical Impact: By resolving nearly 30% of complex, real-world issues (many of which took humans months to resolve), this approach significantly lowers the barrier to entry and maintenance costs for Rust projects.