Technical Summary: TraceView – Interactive Visualization of Agentic Program Repair Trajectories

Problem Statement

Large Language Model (LLM)-based Automated Program Repair (APR) agents generate candidate patches through complex trajectories involving reasoning, tool use, and environmental feedback. While the final outcome of a repair attempt (success or failure) is observable, the intermediate process remains opaque. A failing patch does not reveal whether the agent localized the bug, ignored feedback, repeated actions, or diverged from the task. This lack of visibility makes agentic repair failures difficult to diagnose, reproduce, and prevent.

Existing approaches to trajectory analysis often rely on manual inspection of raw logs or spreadsheets. Prior work, such as Bouzenia and Pradel's manual labeling of 14,000 Thought-Action-Result (TAR) pairs, highlights the high cost and effort required to annotate these trajectories. Without dedicated tooling, researchers struggle to create, filter, and inspect semantic relation labels, making it difficult to connect annotations back to the original evidence in the trace.

Methodology: TraceView

To address these challenges, the authors present TraceView, an interactive tool designed for labeling and visualizing APR agent trajectories. The system is implemented as a Streamlit application and operates through a structured workflow:

1. Data Ingestion and Normalization

TraceView accepts raw inputs (e.g., SWE-agent JSON, JSONL, plain text logs) and pre-labeled inputs (bundled datasets like AutoCodeRover). It normalizes these inputs into ordered Thought, Action, and Result (TAR) steps, creating a unified underlying representation regardless of the source format.

2. Labeling Workflow

The tool supports a two-stage labeling process:

• Action Categorization: Users assign categories to each step (e.g., Explore, Locate, Search, Reproduce, Generate fix, Run tests, Refactor, Explain).
• Semantic Relation Labeling: Users label the relationships between source and target TAR components. Supported relation labels include Alignment, Follow-up, Refinement, Redundancy, Repetition, Divergence, Contradiction, Informative, Triggering, No influence, Misalignment, and Misinterpretation.
• Interface: The labeling workspace presents evidence text from both source and target components. Users can inspect longer evidence via popovers to reduce clutter while maintaining access to the original text.

3. Visualization and Analysis

TraceView renders labeled trajectories as directed graphs with two coordinated views:

• Iteration View: A high-level summary where each node represents a step, labeled by iteration number and action category.
• Detailed TAR View: Expands each step into distinct Thought (T), Action (A), and Result (R) nodes.
  • Node Styling: Thought and Result nodes use fixed colors; Action nodes are colored by their assigned category.
  • Edge Families: Semantic edges connect nodes based on five relation families (e.g., Thought→Action within a step, Result→Thought across steps). Structural edges preserve chronological flow.
  • Filtering and Metrics: Users can filter the graph by relation types (e.g., isolating Repetition loops or Divergence shifts). The tool provides aggregate relation counts and patch outcome summaries (e.g., RESOLVED, APPLIED, GENERATED).
  • Inspector Panel: Selecting a node reveals the underlying text and connected labeled relations, adhering to a "details on demand" principle.

Key Contributions

The paper makes three primary contributions:

1. TraceView Tool: An interactive system for labeling and visualizing APR agent trajectories as structured Thought, Action, and Result graphs.
2. Analysis Workflow: A comprehensive workflow featuring semantic edge labeling, relation filters, patch result summaries, relationship metrics, and node-level evidence inspection.
3. Preliminary User Study: An evaluation involving five researchers that assesses the tool's utility in scanning trajectories and understanding repair workflows, while identifying specific areas for interface improvement.

Evaluation Results

The authors conducted a survey-based user study with five computer science researchers. Participants performed tasks ranging from labeling actions to analyzing graph views.

• Quantitative Findings: Participants rated TraceView positively across all three evaluated dimensions:
  • Labeling Workflow: High ratings for clarity of sections (Median=4) and input formats (Median=5).
  • Evidence and Interpretation: Row-level popovers received a median score of 5 for providing sufficient evidence. Graph interpretability received a median of 4.
  • Analysis Workflow: Participants rated the tool as more efficient than raw logs (Median=5), found graphs easier to scan (Median=5), and found the overview-to-detail hierarchy useful (Median=5).
• Qualitative Feedback:
  • Strengths: Users found the graph visualization effective for understanding the repair process and the overview-to-detail workflow helpful for orientation.
  • Limitations: Some participants noted that action previews were sometimes too short to provide context or too long to scan efficiently. Users requested that node labels include short summaries rather than just identifiers (e.g., "T0"). Additionally, diagonal edges and certain relation labels (e.g., Misalignment) were sometimes difficult to interpret without further explanation.

Significance and Claims

The paper positions TraceView as a necessary step toward making agentic repair failures diagnosable. By moving beyond final patch outcomes to visualize the process of repair, the tool allows researchers to identify where agents align with useful feedback, repeat actions, diverge from tasks, or misinterpret context.

The authors modestly claim that their preliminary study demonstrates the tool's ability to improve trajectory scanning compared to raw logs and supports the utility of an overview-to-detail workflow. They do not claim to have solved the problem of agentic repair entirely but rather to have provided a tool that facilitates the "fine-grained analysis" required to understand and potentially correct agentic behavior. Future work is planned to incorporate user feedback (shorter summaries, hover previews) and conduct larger studies with more diverse trajectories.