💬 NLP

Online Monitoring and Corrective Steering of Programming Agents

This paper introduces LivePlan, a cost-effective framework that enhances the performance of programming agents on complex GitHub issues by employing a deterministic monitor to detect behavioral drifts in real time and selectively consulting an LLM advisor for targeted corrections, thereby achieving significant resolution rate improvements with minimal overhead.

Original authors: Shuyang Liu, Saman Dehghan, Ji Young Kim, Jatin Ganhotra, Martin Hirzel, Reyhaneh Jabbarvand

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shuyang Liu, Saman Dehghan, Ji Young Kim, Jatin Ganhotra, Martin Hirzel, Reyhaneh Jabbarvand

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers can write their own software, fixing bugs and building new features all by themselves. This is the realm of "AI agents," digital workers that use Large Language Models (LLMs)—the same kind of smart brainpower behind chatbots—to read code, figure out what's wrong, and try to fix it. But here's the catch: these digital workers aren't perfect. Sometimes they get distracted, wander off the path, repeat the same mistake over and over, or give up before they're done. It's like sending a very smart but easily confused robot into a giant library to find a specific book; it might start reading the wrong section, get stuck in a loop trying to open a door that's locked, or decide to leave before finding the book. Researchers care deeply about this because if we want AI to handle complex, real-world jobs like fixing massive software projects, we need to make sure these agents don't just drift away from their goals or waste time and money on dead ends.

Enter LIVEPLAN, a new system designed to act like a vigilant, real-time coach for these coding robots. The researchers, Shuyang Liu and their team, noticed that previous attempts to fix these wandering agents had a major flaw: they often tried to "judge" the robot's work and "give advice" at the same time using a single AI brain. This was like asking a single referee to both blow the whistle for a foul and immediately tell the player exactly how to play the rest of the game. The problem? The referee might get confused, invent a foul that didn't happen, and give bad instructions that actually make the player mess up even more.

LIVEPLAN solves this by splitting the job into two distinct roles. First, it uses a deterministic monitor—think of this as a strict, rule-following traffic cop. This cop doesn't guess or hallucinate; it simply watches for specific, clear signs of trouble, like the robot taking the same step twice in a row, skipping a crucial safety check, or staring at the same wall for too long. If the traffic cop sees a problem, then it calls in a smart advisor (a powerful AI) to give a quick, high-level tip on how to get back on track. The key is that the advisor only speaks up when the traffic cop says, "Hey, we have a problem here," rather than constantly interrupting the robot with unsolicited advice.

The team tested this system on real-world GitHub issues, which are like to-do lists for fixing bugs in huge software projects. They found that LIVEPLAN was a game-changer. Compared to the standard "vanilla" agents that just run without a coach, LIVEPLAN helped solve significantly more problems—boosting success rates by up to 15.2% on the hardest tasks, with an average improvement of 9.9%. Perhaps most impressively, it did this while only adding a tiny cost of about $0.08 per fix.

The paper also shows what happens when you don't use this careful approach. They tested other methods that tried to re-plan the robot's entire journey from scratch or checked in too frequently. These often backfired: the "re-planning" coaches sometimes invented problems that didn't exist and sent the robot on wild goose chases, while the "frequent check-ins" often gave advice too late to be useful. LIVEPLAN's approach of waiting for clear signals before intervening proved to be the sweet spot. It successfully guided the agents through the tricky middle and hard problems that usually stump them, without messing up the ones they were already good at. In short, by letting a simple rule-checker watch the door and only calling in the big brain when necessary, the researchers found a way to keep these digital workers focused, efficient, and much more likely to finish the job.

Technical Summary: LIVEPLAN – Online Monitoring and Corrective Steering of Programming Agents

Problem Statement

Fixing GitHub issues in large-scale software projects is a long-horizon task that often requires agents to traverse complex trajectories. Current LLM-based agents frequently suffer from behavioral drift and inefficiencies, such as skipping critical phases (e.g., validation), repeating failed actions, or oscillating between thoughts without progress. These issues lead to task failures or inflated costs even when the agent eventually succeeds.

Existing approaches to correct these drifts suffer from three primary limitations:

Conflated Judging and Advising: Prior methods often use a single LLM to both diagnose trajectory issues and recommend corrections. This creates a conflict of interest where the "advisor" is incentivized to find problems (sycophantic bias), potentially generating misleading advice that derails a trajectory that was actually on track.
Inefficient Triggering: Approaches that rely on periodic LLM evaluation (e.g., every $N$ steps) or post-hoc analysis often provide feedback too late for the agent to recover or incur excessive costs by re-running entire trajectories.
Reliability of LLM Judges: Using an LLM as a judge introduces unreliability; they may miss real problems or flag spurious ones due to hallucination, lacking the determinism required for robust intervention.

Methodology: LIVEPLAN

The paper proposes LIVEPLAN, an online monitoring and intervention framework that decouples judging from advising. It is implemented on top of the SWE-agent framework and consists of three core components:

1. Monitor (Deterministic Judge)

Instead of using an LLM, LIVEPLAN employs a deterministic, rule-based monitor to evaluate the agent's trajectory in real-time.

Trajectory Representations: The monitor utilizes two abstract representations:
- GRAPHECTORY: An enriched graph where nodes are distinct agent actions and edges represent chronological execution. It tracks action loops and self-loops.
- LANGUTORY: An abstraction of low-level actions into a sequence of problem-solving phases (e.g., Navigation, Reproduction, Patching, Validation).
Signal Detection: The monitor checks these representations against a configurable database of general signals to detect behavioral drifts. These include:
- Blocking Drifts: Critical violations such as skipping phases (e.g., submitting without patching), premature patching, or oscillation (repeated reasoning/actions without progress). These trigger immediate intervention.
- Non-Blocking Drifts: Inefficiencies like prolonged stagnation in a single phase or repeated actions. These trigger steering advice but do not block execution.
Intervention Processor: If a drift is detected, the processor decides whether to block the current action (for blocking drifts) or append advice to the history (for non-blocking drifts), up to a configurable limit to prevent stalling.

2. Advisor (LLM-Based Correction)

The LLM-based Advisor is consulted only when the Monitor detects a drift.

Input: It receives the issue description, a slice of the recent trajectory (since the last advice), the latest advice (if any), and the predefined advice corresponding to the detected drift.
Output: It generates a high-level, next-step correction rather than a global re-plan. This keeps the intervention focused and prevents over-constraining the executor.
Cooling Mechanism: To manage cost and latency, the Advisor is subject to a "cooldown" period ( $\theta_c$ ), ensuring it is not invoked for every single step even if drifts are frequent.

3. Workflow

The system operates in a loop: the agent proposes a step $\rightarrow$ the Monitor tentatively updates the trajectory representations $\rightarrow$ rules are checked. If no drift is found, the step executes. If a drift is found, the Advisor is consulted (if cooldown allows), and the resulting advice is injected into the agent's context to steer the next step.

Key Contributions

Technique: The introduction of LIVEPLAN, which separates the deterministic detection of behavioral drifts from the LLM-based generation of corrective advice. This architecture avoids the hallucination and bias inherent in using a single LLM for both tasks.
Empirical Evaluation: A large-scale evaluation across two benchmarks (SWE-bench Verified and SWE-bench Pro) using five diverse LLMs (three executors, two advisors). The study generated 7,752 trajectories, providing a dataset for future research.
Process-Centric Analysis: Beyond resolution rates, the paper analyzes trajectory compliance, drift prevalence, and strategy divergence, offering insights into how interventions improve agent performance.

Experimental Results

LIVEPLAN was evaluated against vanilla SWE-agent, SAGE (a re-planning baseline), and ablated variants (Periodic Advisor, Predefined Advice).

Resolution Rates: LIVEPLAN consistently outperformed all baselines.
- On SWE-bench Pro, it achieved gains of up to 15.2% (average 9.9%) over vanilla agents.
- On SWE-bench Verified, it achieved gains of up to 11.2%.
- Improvements were most pronounced on medium and hard difficulty instances.
Cost Efficiency: The additional cost per instance was minimal, averaging $0.08. In some cases (e.g., MiniMax-M2.5), LIVEPLAN actually reduced overall costs by shortening inefficient trajectories.
Comparison to SAGE: LIVEPLAN significantly outperformed SAGE, which often failed because its global re-plans were factually incorrect (hallucinated file paths or methods), causing agents to fail immediately. LIVEPLAN's targeted, next-step advice proved more robust.
Comparison to Periodic Advisor: While periodic advisors trigger frequently, they often provide misleading advice. LIVEPLAN triggered the Advisor less often but with higher precision, resulting in better resolution rates.
Robustness: Re-evaluation on a subset of instances with deterministic outcomes confirmed that the gains were not artifacts of run-to-run non-determinism.

Significance and Claims

The paper claims that LIVEPLAN represents a significant step forward in online steering of programming agents by addressing the fundamental instability of using LLMs as judges. By relying on deterministic rules to detect when an agent is in trouble and reserving LLMs only for what to do next, the system achieves:

Higher Reliability: It avoids the "misleading re-planning" that causes agents to abandon successful trajectories.
Scalability: It incurs minimal computational overhead compared to methods that re-run agents or invoke LLMs at every step.
Generalizability: The approach is model-agnostic and works across different LLM architectures and benchmark difficulties.

The authors conclude that while inference-time intervention is effective, future work should focus on intervention-aware post-training to ensure agents can reliably execute the corrective advice provided by systems like LIVEPLAN.