💻 computer science

Execution Grounded Multiagent Systems for Reliable Backend Code Generation with Large Language Models'

This paper introduces ExecuGraph, a configurable framework demonstrating that execution feedback is the primary driver of improved code generation accuracy in large language models, while showing that adding multi-agent role decomposition provides no measurable benefit over single-agent retry loops despite significantly higher computational costs.

Original authors: Sai Deekshith Lekkala, Jothi Prabha Appadurai, Rohith Reddy Bellibatlu, Manpreet Singh, Rahul Joshi

Published 2026-08-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sai Deekshith Lekkala, Jothi Prabha Appadurai, Rohith Reddy Bellibatlu, Manpreet Singh, Rahul Joshi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very talented, but slightly dreamy, robot to write computer code. This robot is a "Large Language Model" (LLM), which is like a super-smart student who has read almost every book and code snippet in the library. It can write code that looks perfect on paper, but sometimes it makes subtle mistakes that only show up when you actually try to run the program. In the world of software, this is a big deal because a tiny error can crash a whole website or lose data.

For a while, people thought the best way to fix this was to hire a whole team of robot specialists—a "Multi-Agent System." Imagine a project manager, a strict editor, a logic checker, and a code writer all working together. The idea was that if you break the work down and have different robots check each other's work, the final code would be flawless. But there was a nagging question: Was the improvement coming from having a team, or was it just because the robots were allowed to try again after seeing their mistakes? It's like asking if a student gets better grades because they have a study group, or simply because they were allowed to take a second test after seeing their first one. This paper sets out to solve that mystery by building a special testing machine that can isolate these two factors.

The researchers built a clever framework called ExecuGraph, which acts like a Swiss Army knife for testing code-writing robots. They designed it so they could instantly switch between three modes: a "lone wolf" robot that writes code once and stops; a "lone wolf" that gets to try again if it fails; and the full "dream team" of five different robot agents working together. By running the same 164 difficult coding puzzles through these different modes, they discovered something surprising.

The main finding is that letting the robot try again after seeing its errors is the real magic trick, not having a team of specialists. When they gave a single robot the chance to see its mistakes and retry (a process called "execution feedback"), its success rate jumped by a massive 25.6 percentage points. It went from getting about 56% of the problems right to getting over 81% right. That is a huge win!

However, when they added the full team of five extra agents (a planner, a reviewer, an optimizer, etc.) on top of that retry system, the results didn't get any better. In fact, the team version was statistically indistinguishable from the single robot that just got to retry. The "team" version cost about 3.6 times more in computer power and time but didn't produce a single extra correct answer. The researchers also ruled out the idea that the team was just winning because they got to "roll the dice" more times; they proved that simply generating five random guesses without any feedback didn't help much at all.

There was a twist in the story, though. The researchers found a bug in their own testing machine (a "sandbox" that runs the code) that was accidentally rejecting correct code. Once they fixed this bug, the numbers changed, but the main conclusion stayed the same: the retry loop is the hero, and the extra agents are mostly just expensive decoration.

The paper also looked at how this works with different types of robots. On one specific type of robot (a 16-billion-parameter model), the team approach did help with a specific kind of puzzle called "graph problems," boosting success from 70% to 90%. But on other types of puzzles, the team actually did worse, and the overall score stayed the same. This suggests that adding more agents doesn't automatically make a robot smarter; it just changes which problems it can solve.

In the end, the paper suggests that if you want a reliable code-writing robot, you don't need to build a complex organization of five different agents. You just need to give your robot a single, smart loop: write the code, run it, see what broke, and try to fix it. It's a lot cheaper, faster, and just as effective as hiring a whole committee. The "team" approach might still be useful for generating extra reports or explanations, but for the actual job of writing correct code, the simple "try, fail, retry" strategy is the clear winner.

Technical Summary: ExecuGraph – Execution Grounded Multiagent Systems for Reliable Backend Code Generation

Problem Statement

While Large Language Models (LLMs) have become standard tools for automated code generation, reliability remains a critical limitation, particularly for backend code requiring strict data-structure invariants and edge-case handling. Existing approaches often rely on single-agent, single-pass paradigms or multi-agent pipelines that conflate role decomposition (assigning different tasks to different agents) with execution feedback (using runtime test results to guide refinement).

The central research gap identified is the lack of controlled empirical studies that isolate the marginal benefit of multi-agent role decomposition from the benefit of execution feedback. Many prior systems attribute performance gains to complex agent architectures when those gains may simply stem from the presence of an execution loop. Furthermore, textual self-review mechanisms (e.g., "LLM-as-a-judge") are prone to instability and false negatives, often blocking correct code based on paraphrase-sensitive verbal critiques rather than runtime outcomes.

Methodology: The ExecuGraph Framework

The authors present ExecuGraph, a workflow framework designed explicitly as an instrument to decouple execution feedback from multi-agent role decomposition.

Core Architecture

ExecuGraph is formalized as a typed transition system ( $W = (S, A, \rho, D, T)$ ) over a shared state object $S$ , orchestrated via LangGraph. It features:

Shared State: A typed record (GraphState) persisting the problem statement, plan, code, review artifacts, evaluation results, and retry counts.
Agent Layer: Six specialized agents:
- Planner: Generates algorithmic strategy and complexity analysis.
- Code Generator: Produces Python code based on the plan.
- Logical Reviewer: Provides advisory-only static analysis (structured JSON) but does not gate acceptance.
- Evaluator: The sole authority for acceptance; executes code in a subprocess sandbox and applies hybrid testing (deterministic tests or LLM-generated tests).
- Optimizer: Refactors code only after successful execution.
- Explainer: Generates documentation post-acceptance.
Execution Sandbox: A subprocess-isolated environment with strict import policies (preventing sandbox violations for standard libraries like bisect or heapq), resource limits, and sanitized environments.
Decision Predicate: Acceptance is determined exclusively by execution outcomes (s.evaluation.passed = True), not by static review or verbal self-assessment.

Experimental Design

The framework's primary innovation is its ability to collapse into different baselines via configuration toggles within the same codebase:

Single-Agent One-Shot (SO): Generator only, no retries.
Single-Agent Execution-Retry (SR): Generator + Evaluator with retries (Self-Debug style), no Planner/Reviewer.
Multi-Full (MF): All agents enabled with retries.
Ablations: Specific agents (Planner, Reviewer, Optimizer) or the Retrieval layer (RAG) can be disabled to measure marginal contribution.

The study utilizes three datasets:

Internal-30: A curated suite of 30 problems (DP, Graph, Data Structures).
HumanEval: The full 164-problem benchmark.
APPS-Introductory: A 50-problem subset.

Experiments were conducted using open-weight models (Qwen2.5-Coder-7B and DeepSeek-Coder-V2-Lite-16B) hosted locally on a consumer-grade GPU (RTX 4050) to ensure reproducibility at zero API cost.

Key Contributions

Isolation of Mechanisms: ExecuGraph provides the first controlled comparison separating execution feedback from multi-agent decomposition. By collapsing the framework into SO, SR, and MF conditions, the authors demonstrate that the "multi-agent" gain is often a confound of execution feedback.
Advisory-Only Reviewer: The framework explicitly decouples the "critic" role from the "acceptance" authority. The Logical Reviewer is advisory only, preventing the "false-negative" risk where a confident but incorrect LLM judge blocks valid code.
Empirical Diagnostic Study: The paper presents a rigorous diagnostic showing that execution feedback drives the vast majority of performance gains, while adding multi-agent agents often yields negligible or negative returns relative to cost.
Reproducibility and Artifact Correction: The authors released all code, logs, and scripts. Crucially, they identified and corrected a sandbox import-policy defect in their original harness that had falsely rejected correct programs using standard libraries (e.g., bisect). All reported results are re-scored using the corrected sandbox without re-invoking models.
Cost-Correctness Analysis: Detailed quantification of the trade-offs between accuracy, wall-clock time, token usage, and LLM call counts across different configurations.

Results

1. Execution Feedback vs. Multi-Agent Decomposition

On the HumanEval benchmark (164 problems), the study found:

Execution Feedback is Critical: Single-agent execution-retry (SR) outperformed single-shot (SO) by +25.6 percentage points (81.7% vs. 56.1%, $p < 0.0001$ ).
Multi-Agent Decomposition Adds No Measurable Value: Adding five agents on top of the retry loop (MF) resulted in -1.2 percentage points compared to SR (80.5% vs. 81.7%, $p = 0.76$ ).
Sampling Control: A "best-of-5" sampling control (five independent one-shot attempts filtered by the sandbox) at a similar compute budget to MF gained only +0.4 pp over one-shot. This confirms that the gains from SR are due to feedback, not merely increased sampling diversity.

2. Cost Analysis

Single-Retry (SR): Achieved high accuracy (81.7%) with ~1.5 LLM calls and ~13.3s wall-clock time.
Multi-Full (MF): Required ~5.3 LLM calls and ~59.8s wall-clock time (3.6x calls, 7.3x tokens) for statistically indistinguishable accuracy.
Conclusion: For 7B-scale models, the single-retry configuration is the most cost-effective path.

3. Cross-Model and Category Effects

DeepSeek-Coder-V2-Lite (16B MoE): Multi-agent decomposition showed a +20.0 pp gain specifically on Graph problems (90.0% vs. 70.0%) but caused a -10.0 pp drop in Dynamic Programming and Data Structures. The net result was no change in overall accuracy (83.3%). This suggests a redistribution of capability rather than a universal scaling gain.
Planner Distractor Effect: In qualitative analysis, the Planner agent was found to sometimes act as a "distractor" on 7B models. If the Planner emitted a flawed structural constraint (e.g., incorrect base-case initialization), the Generator faithfully adhered to it, and subsequent retries failed to correct the error because the flawed plan was preserved in the context. Removing the Planner in ablation studies actually improved accuracy by 8.0 pp.

4. Retrieval (RAG)

Enabling the retrieval layer (seeding the Planner with algorithmic snippets) yielded no measurable gain (78.7% vs. 82.0% baseline). The authors attribute this to the small scale of the vector store and the fact that 7B code models already possess strong parametric knowledge of standard algorithms.

Significance and Claims

The paper modestly claims that execution feedback is the primary driver of reliability in LLM code synthesis, not multi-agent role decomposition.

The "Negative" Result is Secure: The finding that multi-agent decomposition adds no value over single-agent retry is the most robust claim in the paper because both conditions receive execution feedback, eliminating the confound of test visibility.
Practical Deployment: The authors recommend Single-Retry (SR) as the default deployment configuration for backend code generation at 7B parameter scales. It achieves equivalent accuracy to Multi-Full at roughly 28% of the model calls and 14% of the token cost.
Artifact Value: The authors acknowledge that Multi-Full may still be valuable for generating artifacts (e.g., invariant lists, explanations) useful for software governance, but not for improving raw correctness.
Hazard of Execution Grounding: The paper highlights a critical lesson: execution-grounded evaluation is only as trustworthy as its implementation. A bug in the sandbox (the import policy defect) can silently corrupt results, converting correct programs into failures. The authors advocate for reference-solution controls in any system where the acceptance criterion is code execution.

In summary, ExecuGraph demonstrates that for reliable backend code generation, a simple execution-retry loop is superior to complex multi-agent orchestration in terms of cost-efficiency and accuracy, provided the execution sandbox is correctly implemented.