💻 computer science

From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws

This paper introduces HarnessFix, a trace-guided framework that diagnoses agent failures by analyzing execution traces and harness code to generate targeted, validated patches that significantly improve LLM agent reliability across multiple benchmarks.

Original authors: Mengzhuo Chen, Junjie Wang, Zhe Liu, Yawen Wang, Qing Wang

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mengzhuo Chen, Junjie Wang, Zhe Liu, Yawen Wang, Qing Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but sometimes confused assistant (the LLM Agent) trying to solve complex puzzles, like fixing a broken computer program or planning a trip. This assistant doesn't work alone; it works inside a "control room" built by engineers. This control room is called the Harness.

The Harness is the set of rules, tools, and safety checks that tell the assistant:

What tools it can use (like a screwdriver or a search engine).
What information it can see (like a map or a manual).
How to report its progress.
When to stop and say, "I'm done."

The Problem: The "Black Box" of Failure

Sometimes, the assistant fails. In the past, when things went wrong, engineers tried to fix it by:

Tweaking the Assistant's instructions: "Hey, try to be more careful!" (This is like telling a driver to "drive better" without fixing the broken brakes).
Guessing: Looking at the final result and hoping a random change helps.

The problem is that these methods often miss the real issue. The failure might not be the assistant's fault; it might be that the Harness gave the assistant the wrong tool, hid a crucial error message, or let the assistant quit before finishing the job. Because the assistant's actions happen in a long, messy chain of events, it's hard to pinpoint exactly where the control room broke.

The Solution: HARNESSFIX (The Detective)

The authors of this paper created a new system called HARNESSFIX. Think of it as a forensic detective for the control room. Instead of guessing, it investigates the crime scene (the failed attempt) step-by-step to find the exact broken part of the Harness.

Here is how HARNESSFIX works, using a simple analogy:

1. The "Translator" (HTIR)

First, the detective takes the messy, confusing log of what happened (the "trace") and the code of the control room, and translates them into a clean, organized map called HTIR.

Analogy: Imagine a chaotic crime scene with scattered clues. The detective organizes them into a clear timeline: "At 10:00, the agent asked for a tool. At 10:01, the tool gave an error. At 10:02, the agent ignored the error." This map connects the agent's actions directly to the rules that governed them.

2. The "Diagnosis" (Finding the Culprit)

Next, the detective looks at the map to find exactly where the chain broke.

Analogy: The detective asks: "Did the agent fail because it didn't know the password? Or because the door was locked by the security system (the Harness)?"
HARNESSFIX identifies if the problem was in the Tool Interface (bad instructions), Lifecycle (letting the agent quit too early), or Observability (hiding the error messages). It creates a specific "Flaw Report" for recurring problems.

3. The "Repair Shop" (Scoped Fixes)

Once the flaw is identified, HARNESSFIX doesn't just let anyone rewrite the whole control room. That would be dangerous and might break other things. Instead, it uses a set of Repair Operators.

Analogy: Think of these as specific tools in a mechanic's kit. If the problem is a loose bolt, the mechanic uses a wrench. If the problem is a flat tire, they use a jack. HARNESSFIX only uses the specific "wrench" needed for that specific flaw. It writes a small, precise patch to fix only that broken part, ensuring it doesn't accidentally break the engine.

4. The "Safety Inspector" (Validation)

Before the new patch is installed, a safety inspector checks it.

Analogy: The inspector asks two questions:
1. "Did this fix the specific problem we found?"
2. "Did this fix accidentally break something else that was working fine?"
If the answer to the second question is "Yes," the patch is rejected. This prevents the system from getting worse while trying to get better.

What Did They Find?

The researchers tested this detective system on four different types of difficult tasks (fixing software, using command lines, doing research, and automating apps).

The Results: HARNESSFIX improved the success rate of the agents by 15% to 50% compared to the original setup.
The Comparison: It performed better than:
- Human experts who manually designed the control rooms.
- Other AI systems that tried to fix themselves by just changing instructions or guessing.
The Discovery: They found that many failures weren't because the AI was "dumb," but because the "control room" had hidden flaws, like hiding error messages or letting the AI quit too soon. HARNESSFIX found these hidden flaws and fixed them.

In Summary

HARNESSFIX is a system that treats a failed AI attempt like a crime scene. It doesn't just blame the AI; it investigates the environment the AI was working in, finds the specific broken rule or tool, and applies a precise, safe fix. This makes the AI much more reliable without needing to retrain the AI's brain or guess what went wrong.

Technical Summary: From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws

1. Problem Statement

LLM-based agents increasingly rely on harnesses—runtime infrastructures providing execution environments, tool interfaces, context management, lifecycle orchestration, observability, verification, and governance (summarized by the ETCLOVG taxonomy). While existing self-improving agents and automatic harness evolution methods attempt to improve performance through runtime supervision, prompt optimization, or workflow search, they often fail to effectively diagnose the root causes of failures.

Current approaches suffer from three primary limitations:

Fragmented Evidence: Failure evidence is scattered across heterogeneous, language-heavy, and non-deterministic execution trajectories, making precise diagnosis difficult.
Mapping Gap: Even when a failure is localized within a trajectory, mapping it back to the specific static harness implementation (source code, prompts, configurations) responsible for it is challenging because runtime behavior and static artifacts are often not explicitly aligned.
Unscoped Modifications: Without targeted causal diagnosis, improvements are often broad, indirect, or poorly scoped. Furthermore, modifying harness mechanisms to fix one flaw risks introducing regressions elsewhere, a challenge exacerbated by the implicit dependencies in agent systems.

Existing methods often optimize based on final outcomes without identifying where in the trajectory the evidence lies or which harness layer failed, leading to superficial fixes that do not address underlying structural flaws.

2. Methodology: HARNESSFIX

The paper proposes HARNESSFIX, a trace-guided framework designed to diagnose agent failures and repair harnesses through a four-agent pipeline. The core philosophy is to drive harness repair from fine-grained, trace-grounded failure diagnoses.

A. Harness-aware Trace Intermediate Representation (HTIR)

To address the heterogeneity of raw execution traces, HARNESSFIX compiles raw traces and harness code into a unified HTIR. This representation normalizes fragmented evidence and captures:

Nodes (TraceSteps): Recoverable execution steps (model calls, tool actions, validations) with unique identifiers.
Attributes: Request/response messages, execution status, and artifact/state effects (read/write/unknown).
Links:
- Temporal links: Preserve execution order.
- Input Provenance links: Trace how a request was constructed from prior trajectory content and harness logic (e.g., memory assembly, prompt building).
- Control Flow links: Explain why a step was triggered by the harness controller (e.g., retry, delegate, terminate).
Node-local Diagnostic Evidence: A localized view for each step combining input provenance, control flow, and artifact effects.
Harness Layer Responsibility Facet: Maps each step to the relevant ETCLOVG layers (Execution, Tooling, Context, Lifecycle, Observability, Verification, Governance) based on the attached evidence.

B. Harness Flaw Diagnosis

A Diagnosis Agent processes HTIR to produce structured diagnosis records:

Symptom Localization: Identifies the failure symptom from the final evaluation result.
Evidence Backtracking: Uses temporal, provenance, and control flow links to generate a ranked set of candidate responsible steps.
Candidate Adjudication: Inspects node-local evidence and harness code references to select the final responsible step(s).
Layer Mapping: Assigns implicated ETCLOVG layers to the responsible steps.
Flaw Consolidation: Groups diagnosis records across multiple executions to identify recurring flaw patterns, summarizing them into actionable Flaw Records.

C. Scoped Repair

To prevent unstable edits and architectural breakage, HARNESSFIX employs Repair Operators rather than free-form self-editing.

Mapping: Flaw records are mapped to specific repair operators (e.g., "Tool-schema narrowing," "Loop guarding") distilled from version-change records in agent repositories.
Repair Specification: The selected operator is instantiated into a flaw-specific specification that defines:
- Target and Scope: Concrete harness artifacts to modify.
- Edit Constraints: Forbidden resources (e.g., benchmark data, evaluators).
- Required Behavior: The specific behavior the patch must implement.
- Validation and Risk: Criteria for acceptance, including target flaw reduction and regression bounds.

D. Patch Validation and Harness Memory

A Validation Agent ensures safety and efficacy:

Pre-validation: Checks the candidate diff for conformance to the repair specification and basic syntax.
Validation Set Assessment: Evaluates the patch on a held-out validation set to ensure it reduces the target flaw occurrence ( $\Delta D_{target} \ge \delta_{min}$ ) without introducing unacceptable regressions ( $R_{new} \le r_{max}$ ).
Harness Memory: Records outcomes (accepted or rejected) to inform future iterations, providing examples for similar flaws or explaining rejection reasons.

3. Key Contributions

HARNESSFIX Framework: A trace-grounded, diagnosis-driven framework that explicitly connects failure evidence to harness-aware diagnosis and scoped harness changes, moving beyond outcome-driven prompt evolution.
HTIR: A harness-aware trace representation that normalizes heterogeneous agent trajectories into step-level evidence structures, enabling precise attribution and repair planning.
Empirical Evaluation: Comprehensive evaluation across four benchmarks (SWE-Bench Verified, Terminal-Bench 2.0 Verified, GAIA, AppWorld) demonstrating significant performance gains.
Systematic Analysis: A detailed analysis of accepted repairs revealing recurring harness-flaw patterns across ETCLOVG layers and the specific stress points of different task settings.

4. Experimental Results

The authors evaluated HARNESSFIX on four benchmarks, comparing it against initial harnesses ( $H_0$ ), human-designed harnesses, and self-evolution baselines (GEPA, SCOPE, ReCreate).

Performance Gains: HARNESSFIX improved held-out test performance over initial harnesses by 15.2% to 50.0% across the benchmarks:
- SWE-Bench Verified: 45.0% $\to$ 57.0% (26.7% relative improvement).
- Terminal-Bench 2.0 Verified: 17.6% $\to$ 26.5% (50.0% relative improvement).
- GAIA: 43.3% $\to$ 61.7% (42.3% relative improvement).
- AppWorld: 36.7% $\to$ 42.2% (15.2% relative improvement).
Comparison with Baselines: HARNESSFIX outperformed all self-evolution baselines and the strongest human-designed harnesses on every benchmark. For instance, on SWE-Bench, it solved 57 tasks compared to 48 for the best human-designed harness (Trae-Agent).
Ablation Study: Removing key components significantly degraded performance:
- Prompt-only repair failed to improve Terminal-Bench and yielded minimal gains elsewhere, highlighting the insufficiency of soft guidance alone.
- Without trace-grounded diagnosis resulted in intermediate improvements but trailed the full system, confirming the need for precise step-level attribution.
- Without scoped repair operators led to regressions, demonstrating the necessity of constraining edits to specific operators.
- Without regression-aware acceptance reduced final scores, proving that validation is critical for converting useful candidates into safe harness changes.

5. Significance and Claims

The paper claims that harness repair is a distinct and critical challenge from prompt optimization. The significance of HARNESSFIX lies in its ability to:

Bridge the Runtime-Static Gap: By using HTIR, it effectively maps dynamic execution failures back to static harness implementation artifacts.
Enable Targeted, Safe Repair: The use of scoped repair operators and regression-aware validation allows for deep modifications to runtime mechanisms (tool interfaces, lifecycle guards, observability) without the instability associated with recursive self-editing.
Generalize Across Domains: The framework's success across diverse domains (software engineering, terminal workflows, research QA, and application automation) suggests that trace-grounded harness repair is a generalizable approach to improving agent reliability.
Reveal Systemic Flaws: The analysis uncovers that different task domains stress different harness layers (e.g., AppWorld stresses Verification and Lifecycle, while GAIA stresses Tool Interface and Observability), suggesting that effective agent engineering requires domain-specific harness tuning rather than one-size-fits-all prompt engineering.

The authors conclude that as agents tackle increasingly complex tasks, the harness becomes a primary determinant of system reliability, and HARNESSFIX provides a systematic method to diagnose and repair these infrastructural flaws.