💻 computer science

Phoenix: Safe GitHub Issue Resolution via Multi-Agent LLMs

Phoenix is a multi-agent LLM system that safely resolves GitHub issues from triage to pull-request creation by employing seven layered safety controls and a baseline-aware evaluation strategy, achieving a 75% oracle-resolution rate on a curated SWE-bench Lite slice while maintaining 100% correctness preservation on real-world issues.

Original authors: Kipngeno Koech, Muhammad Adam, Baimam Boukar Jean Jacques, Joao Barros

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kipngeno Koech, Muhammad Adam, Baimam Boukar Jean Jacques, Joao Barros

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, bustling library called GitHub, where millions of people leave sticky notes on books asking for fixes, new features, or pointing out typos. These notes are called "issues." Usually, a human librarian has to read the note, find the exact page in the book, figure out what's wrong, rewrite the text, and then ask a senior librarian to check it before it goes back on the shelf. This is slow and tiring.

Phoenix is a new team of AI robots designed to do this job, but with a very strict rule: Do no harm.

Here is how Phoenix works, broken down into simple concepts:

1. The Team of Specialists (The Six Agents)

Instead of one super-smart robot trying to do everything at once (which often leads to mistakes), Phoenix uses a team of six specialized workers, like a well-oiled assembly line:

The Planner: Reads the sticky note and draws a map. It decides which pages need changing and how to fix them.
The Reproducer (The Detective): Before fixing anything, this robot tries to recreate the problem. It says, "Okay, if I do X, does the book break?" If it can prove the book is broken, it moves on. If not, it skips this step so the team doesn't get stuck.
The Coder: The writer. It takes the Planner's map and actually rewrites the text on the pages.
The Tester: The quality inspector. It runs the book through a machine to see if the new text causes new errors.
The Failure Analyst: The doctor. If the Tester finds a new error, this robot diagnoses why it happened and tells the Coder how to fix it. They get two tries to fix it; if they fail twice, they stop and ask for human help.
The PR Agent: The messenger. Once the fix is ready, it packages everything up and hands it to a human librarian for final approval.

2. The "Safety Net" (Seven Layers of Protection)

The paper emphasizes that AI can be dangerous if it just starts rewriting books randomly. Phoenix has seven "safety guards" to prevent disasters:

The Fence: It won't let the robots write outside the library walls (preventing them from deleting files they shouldn't touch).
The ID Badge: It checks that the robots have valid keys (tokens) so they don't get locked out mid-job.
The Clean-Up Crew: It strips out messy, confusing parts of the sticky notes before showing them to the robots, so they don't get confused by bad formatting.
The "No-Go" Zone: It refuses to touch the library's security system files (workflow files) because messing with those could lock everyone out.
The Stop Button: If the robots get stuck in a loop or keep making the same mistake, the system pulls the plug.
The Solo Worker: Only one book is worked on at a time to prevent the robots from tripping over each other.
The Fresh Key: It automatically refreshes its ID badges before they expire, so the job never stops due to a timeout.

3. The "Before and After" Test (Baseline Awareness)

This is Phoenix's most clever trick. Sometimes, a library book is already broken before the robot touches it.

Old Way: A robot fixes a typo, but the book still fails a test because it was already broken. The robot gets blamed for the failure.
Phoenix Way: Before the robot changes anything, it takes a "snapshot" of the book's current state. After the robot makes changes, it compares the new state to the snapshot.
- If the book was already broken and stays broken (but no new things break), Phoenix says, "Success! We didn't make it worse."
- If the book was working and now it's broken, Phoenix says, "Stop! We introduced a regression."

4. What the Results Show

The researchers tested Phoenix in two ways:

The Practice Run (SWE-bench Lite): They gave Phoenix 24 specific, pre-defined problems. Phoenix solved 75% of them perfectly without breaking anything that was already working.
The Real World Test (42 Real Issues): They let Phoenix loose on 42 actual issues from 14 different real-world projects.
- Safety: Phoenix achieved 100% "Correctness Preservation." This means it never broke a test that was previously passing. It was incredibly safe.
- Success Rate: However, only about half of the fixes were actually the right fix. The other half were "hallucinations" where the robot wrote code in the wrong place (like writing a repair manual in the fiction section). The robot knew how to fix it, but sometimes couldn't find where the problem was.

The Bottom Line

Phoenix is like a very cautious, highly trained apprentice librarian. It is excellent at not making things worse and very good at following a strict process. However, it sometimes struggles to find the exact location of a problem if the description doesn't match the file names perfectly.

The paper concludes that for AI to be useful in the real world, safety must come before speed. Phoenix proves that by using a team of specialized agents and strict safety guards, you can automate software fixes without accidentally breaking the software. The main thing left to fix is helping the "Planner" robot find the right page in the book more often.

Technical Summary: Phoenix – Safe GitHub Issue Resolution via Multi-Agent LLMs

Problem Statement

Open-source repositories on GitHub accumulate thousands of unresolved issues, creating a persistent maintenance bottleneck. While Large Language Models (LLMs) have demonstrated potential in automating software engineering tasks (e.g., Devin, SWE-Agent, AutoCodeRover), optimizing solely for issue resolution rates often compromises safety. Systems that resolve a portion of issues but introduce regressions in others provide uncertain net value in production environments. Furthermore, existing benchmarks often focus on offline performance, neglecting the complexities of production deployment, such as API gateway filtering, token expiry, and pre-existing broken test suites.

Methodology

Phoenix is a multi-agent LLM system designed to resolve GitHub issues from triage through pull-request (PR) creation while prioritizing correctness preservation over raw resolution rates. The system operates via a closed-loop pipeline coordinated by a central Orchestrator and a label-based GitHub webhook state machine.

System Architecture

The pipeline decomposes the workflow into six specialized agents, each corresponding to a stage in a conventional issue-resolution process:

Planner Agent: Receives sanitized issue text, repository file trees, and relevant excerpts. It outputs a structured JSON plan detailing the approach, files to modify, implementation steps, and test strategy. It uses keyword overlap scoring to determine file relevance.
Reproducer Agent (Optional): Attempts to generate a synthetic test that fails on the base branch to demonstrate the bug. Failure in this step is non-blocking, allowing the pipeline to proceed to coding even if a test harness cannot be automatically generated.
Coder Agent: Generates complete file contents for changed files based on the plan and any prior failure feedback. It enforces self-verification and adheres to existing code styles.
Tester Agent: Executes the project's test suite. Crucially, it employs a baseline-aware test evaluation strategy:
- It stashes Phoenix's changes and runs the suite on the unmodified base branch to collect baseline failures ( $B$ ).
- It restores changes and re-runs to collect post-change failures ( $P$ ).
- Correctness is preserved if $P \setminus B = \emptyset$ (no previously passing test now fails). This allows evaluation on repositories with broken CI pipelines.
Failure Analyst Agent: If new failures are detected, this agent analyzes the output and provides structured feedback to the Coder for up to two retry cycles.
PR Agent: Composes and opens a pull request referencing the issue, transitioning the state to ai:review for human inspection.

Safety Mechanisms

Seven layered safety controls were derived from empirical failure modes observed during development and deployment:

Path-traversal prevention: Validates that file writes do not escape the repository root.
Label-state exclusivity: Ensures atomic state transitions to prevent stale label accumulation.
Workflow file guardrail: Silently drops writes to .github/workflows/ to prevent permission errors.
Content sanitization: Truncates issue bodies and strips stack traces to avoid API gateway (WAF) 403 responses.
Retry cycle limit: Limits Failure Analyst feedback to two rounds; a no-progress detector terminates loops if changes are identical.
Concurrency serialization: A per-installation lock ensures one issue is processed at a time on shared clones.
Installation token refresh: Proactively refreshes GitHub App tokens before expiry to prevent authentication failures during long runs.

Key Contributions

Architecture: A six-agent pipeline with an optional Reproducer, a label-driven state machine, and explicit design rationale for decomposition, allowing for isolated testing and failure attribution.
Baseline-Aware Evaluation: A testing strategy that distinguishes pre-existing failures from Phoenix-introduced regressions, enabling assessment on repositories with broken CI.
Deployment Lessons: A characterization of real-world hazards (WAF filtering, token expiry, permission boundaries) and the specific mechanisms developed to address them.

Results

The paper presents results from two evaluation settings:

1. SWE-bench Lite (24 Instances)

Evaluated on a curated slice of 24 instances across 12 repositories (3 instances each).

Oracle Resolution: Phoenix achieved a 75% oracle resolution rate (18/24 instances).
Regression Safety: On all oracle-passing runs, there were no pass-to-pass regressions.
Failure Modes: Failures were primarily due to terminal ai:failed states or evaluator time-outs, not safety violations.
Note: The authors explicitly state this curated slice is not directly comparable to full-split leaderboard results due to the production webhook path usage.

2. Pilot on 42 Real Issues (14 Repositories)

Evaluated across 14 repositories spanning Easy, Medium, and Hard difficulty tiers.

Correctness Preservation (CP): Phoenix achieved 100% CP (42/42 issues). No new test failures were introduced relative to the baseline.
Resolution Time: On "Hard" tier repositories, the mean end-to-end resolution time was 122 seconds.
Qualitative Inspection: Manual review revealed a dichotomy in output quality:
- ~50% of PRs were well-targeted fixes modifying existing modules.
- ~50% of PRs placed code at incorrect, invented paths (e.g., src/core/config.py). These passed CP because they introduced no regressions but did not fix the reported bug. This limitation is attributed to the Planner's lexical file ranking rather than code synthesis capabilities.

Significance and Claims

The paper positions Phoenix not as a system that maximizes the number of resolved issues, but as a system that prioritizes correctness preservation and safety in production environments.

Safety as a First-Class Requirement: The authors argue that reliable autonomous code modification requires treating safety as an engineering requirement derived from empirical failure modes rather than speculative design.
Modest Claims on Resolution: The paper acknowledges that 100% correctness preservation does not equate to 100% issue resolution. The gap is driven by localization limitations (the Planner's inability to map issue vocabulary to file paths without semantic retrieval), not code generation.
Production Readiness: Unlike prior work focused on offline benchmarks, Phoenix is designed for real webhooks, real branches, and real PRs, with mechanisms specifically tuned to handle the friction of production deployment (e.g., token expiry, API gateways).

The authors conclude that while semantic retrieval is the necessary next step to address localization failures, the current multi-agent architecture successfully demonstrates that safe, end-to-end issue resolution is achievable with rigorous safety controls and baseline-aware testing.