🤖 AI

Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair

Kozuchi Agent is a language-agnostic, open-weight software repair system that leverages a persistent-state, CI-operated pipeline to achieve state-of-the-art performance on SWE-bench Verified and Multi-SWE-bench benchmarks without fine-tuning, demonstrating that its remaining limitations stem primarily from semantic correctness and selection errors rather than formatting or proprietary model access.

Original authors: Mehdi Bahrami, Kosaku Kimura, Satoshi Munakata, Satoshi Nakashima, Yu Ishikawa, Kosuke Maeda, Nao Soma, Kenichi Kobayashi, Keisuke Miyazaki, Keizo Kato, Shigeki Fukuta, Tatsuo Kumano, Nobutaka Imamura

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mehdi Bahrami, Kosaku Kimura, Satoshi Munakata, Satoshi Nakashima, Yu Ishikawa, Kosuke Maeda, Nao Soma, Kenichi Kobayashi, Keisuke Miyazaki, Keizo Kato, Shigeki Fukuta, Tatsuo Kumano, Nobutaka Imamura, Kevin Musgrave, Shahbaz Abdul Khader, Kwun Ho Ngan, Joe Townsend, Fayas Asharindavida, Matthieu Parizy, Akira Sakai, Yuma Ichikawa, Yang Zhao, Michiaki Takizawa, Taku Fukui, Hiroki Ohtsuji, Wei-Peng Chen, Hiromichi Kobashi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, intricate world of software engineering, code is the foundation upon which modern life is built, yet it is prone to breaking. When a program malfunctions, it generates a report describing the error, and a human developer must then diagnose the problem, locate the flaw within thousands of lines of text, and write a correction that fixes the issue without breaking anything else. This process is slow, difficult, and expensive. Recently, a new generation of artificial intelligence has emerged that can read these reports and attempt to write the fixes itself. These systems, often called agents, act like digital apprentices that can navigate a computer's file system, run tests, and edit code. However, teaching these agents to work reliably over long periods has been a major hurdle. They often get lost in the details, forget what they learned in earlier steps, or make mistakes when trying to use the tools needed to fix the code. The challenge for researchers has been to build a system that keeps these digital workers focused, organized, and accountable, allowing them to solve complex problems without constant human supervision.

A team of researchers from Fujitsu Research has addressed this challenge by creating a system called the Kozuchi Agent. Their work focuses on a specific type of artificial intelligence that does not require special training on new data but instead relies on a carefully designed framework to guide its behavior. The researchers built a digital environment where the agent operates in distinct, clearly defined stages. Instead of letting the agent wander freely through a problem, the system forces it to move through a sequence of steps: first reproducing the error to confirm it exists, then locating the exact source of the trouble, and finally writing and testing the fix. At each stage, the agent must produce a specific result before moving to the next, and the system keeps a permanent record of everything the agent does. This structure prevents the agent from getting confused or repeating the same mistakes, turning a chaotic search into a disciplined, auditable process.

The team tested this system on a large collection of real-world software problems, specifically targeting a set of five hundred issues from the Python programming language. They used a powerful, open-source artificial intelligence model that was not modified or retrained for this specific task. To ensure the results were robust, they ran the agent eight separate times on each problem, generating eight different potential solutions. The system then compared these solutions against one another using a clever selection method. Instead of relying on a hidden answer key to pick the best fix, the system used the tests generated by each of the eight runs to evaluate the others. If a fix passed the tests created by another attempt, it was considered a strong candidate. This approach allowed the system to identify the most reliable solution without needing to know the correct answer in advance.

The results were significant. The Kozuchi Agent successfully resolved three hundred and seventy-four out of the five hundred Python problems, a success rate of nearly seventy-five percent. This performance placed the system among the top performers in the field, ranking twelfth overall when compared to all submissions, but ranking first among systems that use open, publicly available models. The researchers also tested the same system on a different set of problems written in the Java programming language. Despite the differences between the two languages, the system performed with similar consistency, solving forty-one out of one hundred and twenty-eight Java issues. This demonstrated that the framework was not tied to a single language but could be applied broadly to different types of software.

A key finding of the study was that the failures that did occur were not due to the agent writing messy or broken code. In fact, nearly all the patches produced by the agent were applied cleanly to the software. The problems lay in the logic of the fix itself; the agent often wrote a correction that looked correct but did not actually solve the underlying issue. This distinction is crucial because it suggests that the main barrier to improvement is not the format of the code or the ability to use tools, but rather the deep understanding of the problem's meaning. The researchers also found that the system was highly efficient in its operations. By automating the workflow and allowing different parts of the process to be reused, they reduced the amount of human effort required to run these tests from five separate manual steps down to a single automated action.

The study explicitly ruled out the idea that the system's success came from simply guessing or from using a massive, proprietary model that only a few companies could access. The agent used a model with twenty-seven billion parameters, which is large but far smaller than the most powerful models available today, and it achieved its results without any special fine-tuning. The researchers also showed that the system did not rely on luck; the performance was consistent across different runs and different types of software repositories. While the system is not perfect and still struggles with the most difficult semantic problems, it represents a major step forward in making artificial intelligence a reliable partner in software engineering. By combining a structured workflow with a smart selection process, the Kozuchi Agent has shown that open, accessible technology can compete with the most advanced systems in solving real-world coding challenges.

Technical Summary: Kozuchi Agent

Problem Statement

Industrial software engineering teams increasingly require Large Language Model (LLM) agents capable of converting bug reports into correct, test-passing patches. However, operating these agents at benchmark scale introduces significant challenges:

Long-Horizon Execution: Real-world fixes require multi-step processes (reproduction, localization, editing, verification) that consume extensive context and time, often leading to unstructured planning and degraded decision quality.
Tool-Grammar Drift: Different LLM families utilize varying conventions for tool calls and structured actions, causing syntax mismatches that invalidate actions and waste inference turns.
Multi-Cluster Heterogeneity: Research environments often span heterogeneous clusters (GPU inference, VM testing, Docker grading), requiring distinct operational scripts for each backend.
Evaluation Cost: Re-running full benchmark pipelines (e.g., SWE-bench Verified) with Docker-based grading is computationally expensive, making iterative experimentation and CI-driven reruns impractical.

Methodology

The authors present Kozuchi Agent, a language-agnostic, open-weight repair agent and a CI-operated evaluation pipeline designed to address these operational blockers. The system is built upon mini-swe-agent but introduces a rigorous harness architecture.

Core Architectural Components

Phase-Driven Orchestration: The agent operates within a directed graph of eight semantically meaningful phases (e.g., reproduction, localization, fixing, verification). Transitions between phases are explicit, with success or fallback edges defined in configuration. This structure prevents unstructured conversations and ensures auditable state handovers.
Persistent Shared State: Instead of relying solely on conversation history, the system maintains a durable shared memory (filesystem-based) containing scripts, generated tests, traces, and handover memos. This allows state to be resumed and audited across context compression.
Deterministic Tool Sandbox: The agent interacts with a fixed set of software engineering tools: line_trace (dynamic tracing), caller_trace (caller discovery), and line_edit (guarded editing). These tools are injected phase-gated to narrow the action space.
Model-Agnostic Action Interface: The runtime separates the logical action contract from model-specific syntax. A "formatter" role normalizes the LLM's output into a single executable command, allowing the same agent logic to run across different model families without forking the code.
CI-Operated Pipeline: The evaluation process is orchestrated via a CI pipeline (.gitlab-ci.yml) that coordinates inference, grading, and reporting across heterogeneous clusters. It features a "reuse mode" that short-circuits expensive stages (e.g., reusing previously validated artifacts) to reduce operator touch-points from five to one per cycle.

Cross-Agent Test-Time Selection

A key innovation is the Cross-Agent Selector. Instead of relying on a learned verifier or hidden benchmark tests during selection:

Candidate Generation: The system runs $K=8$ independent inference runs of the same model. Each run generates a candidate patch and its own set of bug-revealing (FAIL_TO_PASS) and regression-preservation (PASS_TO_PASS) tests.
Cross-Application: The selector constructs a $K \times K$ matrix where the tests generated by run $j$ are applied to the patch generated by run $i$ .
Scoring: A score is computed based on the pass rates of these cross-applied tests, weighted by a fixed prior ( $w_B=0.3$ for bug-revealing, $w_R=0.7$ for regression). The candidate with the highest score is selected.
Constraint: This process uses only evidence archived by the candidate runs, ensuring no leakage of hidden benchmark test results.

Key Contributions

C1: Auditable Phase-Driven Harness: A model-agnostic agent harness that decomposes issue resolution into eight phases with explicit transitions and a deterministic tool sandbox, making long repair loops inspectable and repeatable.
C2: Reusable CI Pipeline: A multi-cluster benchmark pipeline that reduces operator effort and enables the reuse of expensive evaluation stages, with all empirical values regenerable from public artifacts.
C3: Cross-Agent Selector: A test-time selection mechanism that merges multiple inference runs using cross-applied tests, separating selector regret from instance hardness without requiring hidden test access.
C4: Competitive Open-Weight Results: Empirical evidence showing that a 27-billion-parameter open-weight model, without fine-tuning, achieves state-of-the-art performance among open systems on SWE-bench Verified and Multi-SWE-bench Java.
C5: Cross-Language Transfer: Demonstration that the same harness, phase graph, and selector logic transfer effectively from Python to Java with minimal per-phase behavioral deviation.

Results

Python (SWE-bench Verified)

Performance: Using a locally hosted Qwen3.5-27B model, Kozuchi Agent resolved 374/500 instances (74.80%) on the official cloud evaluator.
Ranking: It ranks 12th overall among 135 cataloged submissions and 1st among open-weight systems.
Statistical Significance: Under Benjamini–Hochberg FDR correction, it significantly outperforms 16 of 17 curated open-weight peers, including a 480B parameter Qwen3-Coder configuration.
Failure Analysis: Of the 126 unresolved instances, 91.3% were "Wrong Fix" (patches applied cleanly but failed hidden tests), indicating the primary failure mode is semantic correctness rather than formatting or application errors.
Selection Lift: The cross-agent selector contributed +14 resolved instances over a size-ordered first-candidate baseline.

Java (Multi-SWE-bench Java)

Performance: The same 27B agent configuration resolved 41/128 instances (32.03%).
Ranking: It ranks 4th overall among 42 submissions and 1st among strict open-weight submissions.
Transfer: Per-phase behavior remained within $\pm 5$ percentage points of the Python run, and the same 27B model outperformed the strongest same-class open-weight Java peer (Qwen2.5-72B) by 38 instances.

Operational Efficiency

The CI pipeline reduced operator touch-points from 5 to 1 per cycle.
6 of 9 CI stages are reusable across iterations.
494/495 produced patches applied cleanly through the harness.

Significance and Claims

The paper positions Kozuchi Agent not as a breakthrough in model capability, but as a significant advancement in harness engineering and operational reproducibility.

Industrial Impact: The primary contribution is the replacement of ad-hoc, manual evaluation workflows with a standardized, auditable, and reusable CI pipeline. This reduces the cost of experimentation and enables rigorous comparison of open-weight models.
Evidence Boundary: The authors explicitly state that their claims regarding the harness (phases, tools, CI) are supported by operational signatures and artifact audits, while the specific gains from candidate count and selection are supported by controlled ablations. They do not claim to have solved the semantic correctness gap entirely, noting that remaining failures are primarily due to the model's inability to generate semantically correct patches rather than formatting issues.
Modesty: The authors emphasize that the system is a "pre-production evaluation" tool. They do not claim production deployment, end-user developer KPIs, or deployment against proprietary internal repositories. The work isolates the value of the harness in enabling open-weight models to compete with proprietary systems, rather than claiming the model itself has surpassed frontier closed models in all aspects.

In summary, Kozuchi Agent demonstrates that with a robust, auditable, and language-agnostic harness, a 27B open-weight model can achieve top-tier performance in automated software repair, provided the evaluation pipeline is designed to maximize candidate diversity and selection rigor without relying on hidden test leakage.