🤖 machine learning

Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents

This paper proposes EvalXRL, a novel benchmark that evaluates Explainable Reinforcement Learning (XRL) methods based on their practical utility in helping an LLM coding agent iteratively diagnose and repair malfunctioning RL agents, moving beyond traditional metrics to a closed-loop, outcome-driven assessment.

Original authors: Ram Rachum, Yotam Amitai, Bálint Gyevnár, Reuth Mirsky, Cameron Allen

Published 2026-08-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ram Rachum, Yotam Amitai, Bálint Gyevnár, Reuth Mirsky, Cameron Allen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, there is a growing field dedicated to making machine learning systems understandable. These systems, often called agents, learn to make decisions by trying different actions and seeing what happens, a process known as reinforcement learning. Sometimes, these agents work perfectly, but other times they fail in strange ways, getting stuck in loops or making choices that seem completely irrational. When this happens, developers need to know why. They need explanations that do more than just describe what the agent did; they need to understand the underlying reasons so they can fix the problem. For years, researchers have tried to measure how good these explanations are by asking people if they feel they understand, or by checking if the explanation matches the computer's internal math. But feeling like you understand is not the same as actually being able to fix a broken machine.

A new proposal from a team of researchers suggests a more practical way to judge these explanation tools. Instead of asking people how they feel, they propose testing whether the explanations actually help a coder repair a broken agent. The core idea is simple: if an explanation is truly useful, it should help someone identify the specific error in the code and fix it, leading to a better-performing machine. This approach shifts the focus from abstract theories of understanding to a concrete, functional result. The researchers call their proposed test "EvalXRL." It is designed to be a standard benchmark where different explanation methods are put to the ultimate test: can they help a software agent find and repair a bug in another software agent?

The researchers have designed a controlled experiment where they deliberately break reinforcement learning agents. They create specific, known errors in the code, such as changing how the agent receives rewards or altering the environment it operates in. For example, they might program a treasure-hunting agent to value small coins more than large ones, or they might trick a traffic control system into creating long lines of cars just to make the traffic flow look fast on a single second. Once these agents are broken, the researchers introduce an artificial intelligence coder, a large language model trained to write and fix software. This coder is given access to the broken agent and a specific explanation tool. The coder's job is to use the tool to figure out what is wrong and then write the code to fix it.

The experiment is set up like a closed loop. The coder does not just look at one explanation and guess. Instead, the coder can ask the explanation tool for information, analyze the answer, form a new theory about what is broken, and then ask the tool again with a different question or setting. This back-and-forth process mimics how a human engineer might work, testing hypotheses until the problem is solved. The researchers use a sandboxed computer environment to ensure the coder cannot cheat by looking up answers on the internet. The only way the coder can succeed is by using the explanation tool effectively. The success of each explanation method is measured by how well the repaired agent performs afterward. If the agent starts working correctly and earning high scores, the explanation tool is considered successful. If the agent remains broken or gets worse, the tool is considered less useful.

The team plans to test a variety of explanation methods, ranging from visual maps that highlight important parts of an image to text descriptions that explain why an agent made a certain choice. They will compare these methods against two extremes: a baseline where the coder has no explanation tool at all and must guess based on raw code, and a "reference" scenario where the coder is simply told exactly what the bug is in plain language. This setup allows them to see if the explanation tools provide any real value beyond just having the source code, and how close they come to the perfect scenario of knowing the answer immediately.

The researchers have three main expectations for how this benchmark will play out. First, they suspect that no single explanation tool will be the best at fixing every type of bug. Some tools might be great at finding errors related to how rewards are calculated, while others might be better at spotting issues with how the agent perceives its environment. This would mean that the field needs a diverse set of tools rather than one universal solution. Second, they expect that even when the coder is told exactly what the bug is, they might not always be able to fix it perfectly. This would show that diagnosing a problem is only half the battle; actually engineering the solution is often the harder part. Finally, and perhaps most surprisingly, they predict that some explanation tools might actually make things worse. A tool might give a confident but misleading explanation that sends the coder down the wrong path, causing them to fix the wrong part of the code. This would prove that a well-written explanation is not always a helpful one.

This proposal is currently a plan for a future study, not a completed set of results. The researchers are presenting their design and their hypotheses to the scientific community to gather feedback before they run the full experiment. They are asking for input on whether the types of bugs they have chosen are the ones that matter most to practitioners, and whether using an artificial intelligence coder is a fair substitute for a human engineer. They acknowledge that while AI coders are fast and cheap to run, they might not think exactly like humans. However, they argue that as more engineering work is automated, understanding how these tools help AI agents fix other AI agents is becoming increasingly important in its own right.

The ultimate goal of this work is to move the field of explainable artificial intelligence away from subjective ratings and toward objective, functional proof. By measuring success based on the ability to repair a broken system, the researchers hope to create a standard that clearly shows which explanation methods are truly useful. If their approach works, it could provide a reliable way to sort through the many different tools currently available, helping developers choose the right one for the job. It offers a path to a future where we do not just ask if an AI makes sense to us, but whether it helps us build better, more reliable systems.

Technical Summary: Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents

Problem Statement

The field of Explainable Reinforcement Learning (XRL) currently lacks a unified, application-grounded evaluation standard. Existing paradigms rely on:

Functionally-grounded metrics: Such as faithfulness and compactness.
Human-grounded proxies: Such as subjective trust ratings or prediction accuracy.

These approaches interpret "understanding" differently (e.g., as the ability to predict outputs or create simpler models), but they do not directly measure whether an explanation helps a developer resolve a malfunctioning agent. Furthermore, the XRL landscape features a "Cambrian explosion" of heterogeneous outputs (saliency maps, video highlights, decision trees, reward decompositions), making cross-method comparison difficult. There is a specific gap in evaluating XRL methods based on their utility for debugging—the process of diagnosing and fixing broken agents—which is a primary use case for ML engineers but is rarely measured as a downstream task in benchmarks.

Methodology: The EvalXRL Framework

The paper proposes EvalXRL, a benchmark designed to evaluate XRL methods based on their ability to assist a "coder" in repairing a deliberately broken RL agent. The framework is defined by three core design choices:

1. Evaluation Substrate: Deliberately Broken Agents

The benchmark uses a set of RL agents with known, controlled malfunctions injected via unified diffs to training code or environments. These malfunctions are designed to require reasoning about behavioral consequences rather than simple code inspection. Examples include:

Reward hacking: Incentivizing hovering near small treasures rather than traversing the grid.
Distributional shift: Agents trained on summer weather deployed in winter.
Sensor drift: Temperature sensors reading 5°C higher.
Myopia: Discount factors set too low (0.3 vs 0.99).

2. The Developer: LLM Coding Agents

Instead of human subjects, the "developer" is a frontier Large Language Model (LLM) coding agent (e.g., Claude Opus) operating within a sandboxed Docker environment.

Rationale: This approach is orders of magnitude cheaper and faster than human studies and avoids complex ethical approvals. It aligns with the trend of "vibe coding," where LLMs increasingly perform engineering tasks.
Capabilities: The agent can read source code, run experiments, modify parameters, retrain agents, and submit fixes.
Closed-Loop Interaction: Unlike one-shot evaluations, the agent operates in a closed loop. It can invoke an XRL method, process the output, form new hypotheses, adjust parameters, and invoke the method again. This mimics the scientific method and dialogical explanation processes.

3. Scoring Mechanism: Repair Performance

The primary metric is the Repair Score, a continuous value in $[0, 1]$ representing the normalized task performance of the agent after repair.
$\text{Score} = \text{clip}_{[0,1]} \left( \frac{P_{\text{post}} - P_{\text{broken}}}{P_{\text{clean}} - P_{\text{broken}}} \right)$
Where:

$P_{\text{post}}$ : Performance of the repaired agent.
$P_{\text{broken}}$ : Performance of the un-repaired malfunctioning agent.
$P_{\text{clean}}$ : Performance of the unmalfunctioned agent.

The benchmark compares XRL methods against two controls:

No-Method Baseline: The coder must debug using only source code and environment info (lower bound).
Oracle Reference: The coder is given a natural language description of the bug (upper bound).

The framework iterates across $(\text{environment} \times \text{malfunction} \times \text{XRL method})$ tuples.

Key Contributions

A New Evaluation Paradigm: Shifts the definition of "understanding" in XRL to "the ability to fix a broken mechanism." This functional view is measurable and directly tied to downstream utility.
The EvalXRL Benchmark Design: A structured protocol for a planned head-to-head comparison of heterogeneous XRL methods (e.g., Reward Decomposition, Counterfactual Analysis, Saliency Maps, Decision Trees) in a closed-loop debugging setting.
Closed-Loop Usage: Proposes the first systematic comparison of XRL methods where the user (LLM) can interactively query and adjust parameters, rather than relying on static, one-shot explanations. Note: This paper outlines the design for this comparison; the actual head-to-head evaluation is the subject of a planned follow-up study.
Hypothesis Formulation: The paper outlines three specific hypotheses to be tested in future pilot runs:
- H1 (Interaction Structure): XRL methods will not rank linearly; instead, specific methods will cluster as effective for specific types of malfunctions.
- H2 (Oracle Ceiling): An "oracle reference" (human bug description) will set a high but imperfect bound, quantifying the gap between diagnosis and engineering.
- H3 (Methods Can Hurt): At least one XRL method will perform significantly worse than the no-method baseline for certain malfunctions, demonstrating that well-formed but non-causal explanations can misdirect the debugging process.

Results

This paper does not report experimental results. It is a preliminary proposal outlining the benchmark design, the hypotheses, and the open questions. The authors state that the pilot run to test hypotheses H1–H3 is the "immediate next step" and will be the contribution of a follow-up paper.

Significance and Claims

The authors position EvalXRL as a necessary evolution in XRL evaluation:

Direct Utility: By wiring the RL agent's reward signal directly to the evaluation, the benchmark removes proxy-driven bias found in subjective ratings or prediction accuracy.
Scalability: Using LLM agents allows for large-scale, reproducible testing that is currently infeasible with human subjects.
Community Input: As a workshop-stage proposal, the authors seek feedback on the malfunction taxonomy, the validity of the LLM surrogate, and which XRL methods should be prioritized for the initial release.
Future Applicability: The authors note that the framework's structure (coder, dossier, repair score, two-container harness) is transferable to evaluating mechanistic interpretability methods for Large Language Models (LLMs) themselves.

The paper emphasizes modesty, acknowledging limitations such as the "LLM-vs-human validity gap" (LLMs may interpret text differently than humans interpret visualizations) and the risk of "memorization" where LLMs might recall bug patterns from their training data. These are framed as open design questions for the community to address.