💻 computer science

SAGE: A Self-Adaptive Agentic Framework for Execution-Guided Code Generation and Self-Repair

This paper introduces SAGE, a self-adaptive multi-agent framework that significantly improves the reliability of generating runnable multi-file projects by employing a closed-loop MAPE-K control cycle to iteratively execute code, analyze runtime errors, and perform self-repair, thereby achieving a statistically significant increase in success rates compared to single-pass generation.

Original authors: Harshil Lodhiya

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Harshil Lodhiya

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but slightly absent-minded robot how to build a complex LEGO castle. You give the robot a simple instruction: "Build a castle with a moat and a drawbridge." The robot, trained on millions of pictures of castles, might instantly snap together a structure that looks perfect. But if you try to pull the drawbridge, it might snap off because the robot used a piece that was discontinued years ago, or it forgot to glue the base to the ground. In the world of computer science, this is the challenge of "Large Language Models" (LLMs). These are AI systems that are incredibly good at writing small pieces of code, like a single function, but they often struggle when asked to build a whole, working software project from scratch. They might forget to install necessary tools, use outdated instructions, or create a program that looks correct on paper but crashes the moment you try to run it.

For a long time, the solution was to just ask the robot to try again, hoping it gets it right the first time. But this paper introduces a smarter approach: instead of just guessing, let the robot actually build the castle, try to open the drawbridge, and if it breaks, fix it immediately. This is the core idea of "execution-guided self-repair." It's like having a quality control inspector who doesn't just look at the blueprints but actually tries to use the product. If the product fails, the inspector tells the builder exactly what went wrong, and the builder fixes it. This cycle of building, testing, and fixing continues until the project works perfectly. The big question researchers have been asking is: Does this "try, fail, fix" loop actually make the AI much better at building complex software, or does it just make the robot spin its wheels and get confused?

The researcher behind this study, Harshil Lodhiya, created a new system called SAGE (Self-Adaptive Agentic Framework for Execution-Guided Code Generation and Self-Repair) to answer that question. Think of SAGE as a tiny, automated construction crew with four distinct roles working in a loop. First, a Planner breaks down your big goal into a detailed checklist. Second, a Coder builds the files based on that list. Third, a Validator acts as the strict inspector: it sets up a clean, isolated workshop, installs the necessary tools, and tries to run the program. If the program crashes or fails to do what it was supposed to do, the Validator doesn't just give up; it sends the error message to a fourth member, the Feedback Agent. This agent acts like a detective, reading the error report and telling the Coder exactly which file to change and how to fix it. The Coder then makes the repair, and the whole team tries again. This cycle repeats until the project runs successfully or they run out of attempts.

The team tested SAGE on a special set of 15 tricky tasks they designed specifically to trip up AI, such as using old library commands that no longer exist or missing a required output file. They compared SAGE's "fix-it" loop against a standard AI that just tries to build the project once and hopes for the best. The results were clear: the self-repair loop made a huge difference. When using a strong AI model (gpt-4.1-mini), the standard "one-shot" approach succeeded in about 69.3% of the cases. But when they added the SAGE repair loop, the success rate jumped to 92.0%. That is a 22.7 percentage point improvement, which the researcher says is a significant and meaningful gain. Essentially, the loop rescued tasks that the AI would have otherwise failed completely, such as fixing code that tried to use a "month" frequency alias that had been removed from a popular data library.

However, the paper also tested a more aggressive version of the idea. They wondered: what if the AI didn't just fix crashes, but also checked if the program was perfectly complete, even if it didn't crash? For example, if a program was supposed to generate a report file but finished without making an error, just without the file, would the AI catch that? They built a "correctness-aware" system to do this. Surprisingly, this extra strictness didn't help much. The success rate only went up by a tiny 2.7 points (to 94.7%), and the researcher found this difference wasn't statistically significant. In fact, the extra checking sometimes caused the AI to "over-fix" things, breaking programs that were already working fine. The lesson here is that while telling the AI exactly what success looks like is helpful, constantly forcing it to re-check and re-fix things can actually make it worse.

In short, the paper shows that giving AI a "try, fail, fix" loop is a powerful way to build working software, turning a success rate of about 69.3% into 92.0%. It proves that letting the AI see its own mistakes in a real, isolated environment is the key to unlocking its potential for complex projects. But it also warns us that being too perfectionist can backfire; sometimes, it's better to let the AI fix the big errors and leave the rest alone. The researcher has made their code and their test suite open for anyone to use, hoping to help others build software that can heal itself.

Technical Summary: SAGE – A Self-Adaptive Agentic Framework for Execution-Guided Code Generation and Self-Repair

Problem Statement

While Large Language Models (LLMs) have achieved high competence in synthesizing single, self-contained functions (e.g., HumanEval, MBPP), they remain unreliable when tasked with generating complete, multi-file software projects. Open-loop generation (single-pass output) frequently produces artifacts that parse correctly but fail at runtime due to:

Missing or unresolvable dependencies: Packages not declared or installed.
API version drift: Usage of deprecated or removed library functions (e.g., pandas.np, matplotlib.cm.get_cmap, or the 'M' frequency alias in pandas) due to training data lagging behind current releases.
Silent incompleteness: Programs that exit with code 0 but omit required output artifacts or steps.
Type and schema errors: Mismatches in multi-stage data flows.

Current self-refinement and self-debugging approaches often operate at the granularity of single functions or competitive programming solutions graded by hidden unit tests. They largely fail to address the engineering complexities of provisioning isolated environments, resolving dependencies, and driving a multi-file artifact to a successful, zero-exit-code execution.

Methodology: The SAGE Framework

SAGE is an open-source, multi-agent framework that treats project synthesis as a closed-loop, self-adaptive control problem, explicitly modeled after the MAPE-K (Monitor, Analyse, Plan, Execute over shared Knowledge) reference model for autonomic computing.

Architecture and Agents

The system operates via a deterministic orchestrator driving a generate → validate → diagnose → repair cycle:

Planner: Decomposes a natural-language goal into a structured plan. This includes a project description, file specifications (paths, dependencies), a list of PyPI requirements, an entry point, and a success_criteria block (defining expected artifacts and stdout markers). The planner enforces invariants: the project must be runnable via python <entry_point> without external input files (synthesizing sample data if needed) and must exit with code 0.
Coding Agent: Materializes the plan. On the first iteration, it generates all files. In subsequent iterations, it applies structured repairs (modify, create, delete) or regenerates files based on feedback.
Validator: Acts as the Monitor. It executes the project in an isolated Python virtual environment (venv) to ensure dependency isolation. It performs:
- Syntax checking (py_compile).
- Dependency installation (pip).
- Execution with a timeout, capturing stdout, stderr, and return codes.
- Optional Correctness-Aware Oracle: Checks if the run produced expected artifacts and stdout markers, not just a zero exit code.
Feedback Agent: Acts as the Analyser and Planner. It converts unstructured runtime tracebacks into a typed, structured repair contract (FeedbackResult). This includes a root-cause analysis, specific file edits, and dependency changes (additions/removals). This structured interface prevents free-form regeneration and allows the orchestrator to deterministically apply changes.

Execution Loop

The loop continues until the project runs successfully (zero exit code, no errors, and optionally meeting success criteria) or an iteration budget (default 5) is exhausted. The framework is provider-agnostic, supporting any OpenAI-compatible endpoint.

Evaluation Methodology

The author introduces SAGE-Bench-v2, a benchmark of 15 self-contained, offline Python tasks specifically designed to induce real-world failure modes (API drift, silent incompleteness, type errors) rather than textbook algorithmic exercises.

Independence: Tasks are authored for this work to avoid training data contamination.
Oracle: Success is determined by an independent re-execution oracle that re-runs the generated project in a fresh environment, checking exit codes, artifact existence, and stdout tokens.
Metrics: Uses the unbiased Pass@k estimator (Chen et al., 2021). Statistical significance is assessed via paired Wilcoxon signed-rank tests with Vargha–Delaney effect sizes.

Key Results

The study evaluates SAGE against an open-loop baseline on gpt-4.1-mini:

Effectiveness of Execution-Guided Repair (RQ1):
- Closing the loop significantly improves project-level Pass@1 from 69.3% (baseline) to 92.0% (+22.7 points, $p=0.042$ , medium effect).
- The framework achieves a 100% solve rate at Pass@3, rescuing tasks that open-loop generation fails completely (e.g., fixing the deprecated pandas 'M' frequency alias).
- Repair specifically targets real runtime and dependency failures that the model cannot observe without execution.
Specifying vs. Enforcing Success (RQ2):
- A correctness-aware oracle (which repairs based on planner-specified success criteria, not just exit codes) improves Pass@1 to 94.7%, but this is not statistically significant compared to the exit-code oracle ( $p=0.50$ , negligible effect).
- While the correctness-aware oracle successfully rescues "silent incompleteness" cases, the additional repair churn it triggers causes regressions in other tasks (e.g., breaking previously working code).
- Conclusion: Specifying success criteria helps, but aggressively enforcing them via additional repair loops does not significantly improve net success on strong models.
Model Capability Dependence (RQ3):
- On a weaker model (gpt-3.5-turbo), execution-guided repair still yields a significant improvement (Pass@1 from 60.0% to 75.0%), suggesting the benefit persists across model tiers.

Significance and Claims

The paper claims SAGE provides a practical framework for self-healing synthesis and offers rigorous evidence regarding the limits of self-repair:

Control-Theoretic Framing: By adopting the MAPE-K vocabulary, the paper clarifies the design of LLM-based code generation as an autonomic system, highlighting where feedback controllers succeed (observable runtime errors) and where they face blind spots (silent logical failures).
Empirical Limits of Repair: The work challenges the notion of self-repair as a "silver bullet." It demonstrates that while execution feedback is essential for fixing dependency and version drift, more aggressive enforcement (via correctness-aware oracles) can be counterproductive due to the stochastic nature of LLM edits, which can introduce new defects while fixing old ones.
Specification over Enforcement: The key takeaway is that specifying success (via the Planner) is more valuable than aggressively enforcing it (via the Validator's repair loop).
Benchmark Rigor: The paper contributes a contamination-free benchmark and a reproducible harness that grades success via independent re-execution, addressing the issue of optimistic self-assessment in prior works.

The author concludes that the practical value of closed-loop synthesis will likely grow as the gap between training data and deployed APIs widens, but future work must address the trade-off between repair depth and the risk of regression.