Process-Centric Analysis of Agentic Software Systems

This paper introduces Graphectory, a graph-based framework for analyzing the stochastic execution trajectories of agentic software systems, which reveals that richer prompts and stronger models yield more complex reasoning patterns while enabling real-time monitoring and intervention that significantly improves problem resolution rates and efficiency.

The Gist

Imagine you are hiring a team of brilliant but very new interns to fix a massive, complex software bug in a company's codebase.

In the old way of doing things (which the paper calls "Outcome-Centric"), you would only look at the final result. Did they hand you a fixed file? Yes? Great, they get a gold star. No? They get fired. You don't care how they got there. Did they spend three days staring at the wrong file? Did they delete the wrong code and then try to put it back? Did they panic and run in circles? If the final file was fixed, you never knew.

This paper introduces a new way to watch these "AI Interns" (called Agentic Systems) work. It's called "Process-Centric Analysis." Instead of just checking the final homework, the authors built a tool called Graphectory to watch the entire movie of how the AI thinks and acts.

Here is a simple breakdown of the paper's ideas using everyday analogies:

1. The Problem: The "Black Box" of AI

Current AI agents (like SWE-agent or OpenHands) are like magic boxes. You give them a problem, and they give you a solution. But inside the box, they might be:

• Wandering aimlessly: Looking in the wrong drawers for a tool.
• Spinning in circles: Trying the same fix over and over when it doesn't work.
• Skipping steps: Fixing the code but forgetting to test it.

If they get lucky and fix it, we celebrate. If they fail, we just say "it didn't work." We miss the chance to learn why they struggled.

2. The Solution: Graphectory (The "Flight Recorder")

The authors created Graphectory. Think of this as a flight recorder (black box) for the AI's brain, but instead of just recording "crash" or "safe landing," it maps out the entire flight path.

• The Map: It turns the AI's linear list of actions (Step 1, Step 2, Step 3...) into a 3D map.
• The Connections: It draws lines between actions.
  • Time Lines: "I looked at the file, then I edited it."
  • Structure Lines: "I looked at the main folder, then I went deeper into a sub-folder."
• The Result: You can instantly see if the AI is taking a direct highway to the solution or if it's stuck in a traffic jam of loops, backtracking, and wrong turns.

3. The "Language" of the Journey: Langutory

Looking at a giant map can be overwhelming. So, the authors also created Langutory.

• Analogy: Imagine the AI's journey is a long sentence. Langutory is the summary of that sentence.
• Instead of reading 50 steps, Langutory says: "The agent spent 5 minutes Looking (Localization), then 2 minutes Fixing (Patching), then 1 minute Checking (Validation)."
• This lets researchers quickly spot if an AI skipped the "Checking" part or spent 90% of its time just "Looking" without ever fixing anything.

4. What They Discovered (The "Aha!" Moments)

The team watched 4,000 of these AI journeys and found some surprising things:

• Success isn't always efficient: Even when the AI fixed the bug, it often took a very long, winding road. It was like a driver who finally arrived at the destination but drove through three different states and got lost twice along the way.
• The "Smart" AI gets lost more: Surprisingly, the "smarter" AI models (the ones with bigger brains) often explored more and took longer paths. They were like detectives who read every single book in the library before finding the clue. While this sometimes helped them solve harder problems, it was often a waste of time.
• Failure looks messy: When the AI failed, the map showed chaotic patterns: lots of loops (spinning in circles), going back and forth between files, and trying to fix things that weren't broken.

5. The Superpower: Real-Time Intervention

The coolest part of the paper is that they didn't just watch the movies; they built a Live Coach.

• How it works: As the AI is working, the Graphectory system watches in real-time.
• The Intervention: If the system sees the AI starting to spin in circles or skip a crucial step (like testing), it pauses the AI and sends a diagnostic message: "Hey, you've been looking at this file for 10 minutes without changing anything. You might be stuck. Try running a test instead."
• The Result: When they used this "Live Coach," the AI fixed more bugs (up to 23% more) and did it much faster. It was like having a GPS that says, "You're going in circles, turn around now," instead of waiting until the driver crashes.

Summary

This paper is about moving from asking "Did it work?" to asking "How did it work?"

By turning the AI's messy thought process into a clear map (Graphectory) and a simple summary (Langutory), the researchers can spot bad habits, fix them while the AI is working, and build smarter, more efficient software agents for the future. It's the difference between judging a chef only by the taste of the final dish versus watching them chop, cook, and season to see where they can improve their technique.

Technical Summary

Here is a detailed technical summary of the paper "Process-Centric Analysis of Agentic Software Systems".

1. Problem Statement

Agentic software systems, powered by Large Language Models (LLMs), are increasingly used for complex tasks like software engineering. However, current evaluation methods are predominantly outcome-centric, focusing solely on whether an agent successfully resolves a task (e.g., fixing a bug). This approach has significant limitations:

• Lack of Insight: It fails to explain how the agent reasoned, planned, or acted.
• Hidden Inefficiencies: It masks recurrent inefficiencies, such as repetitive loops, backtracking, or strategic missteps, which may lead to success by chance rather than systematic reasoning.
• Scalability Issues: Existing attempts to analyze agent trajectories rely on manual, expert-driven taxonomies of failure modes, which are biased, incomplete, and do not scale to thousands of trajectories.
• Linear Limitations: Raw trajectory logs are linear sequences that fail to capture the semantic structure of execution flow, problem-solving strategies, or the relationships between actions.

2. Methodology

The authors propose a novel framework to systematically encode and analyze the temporal and semantic relations in agentic workflows.

A. Core Data Structures

1. Graphectory: A cyclic directed graph ($G = (V, TE \cup SE)$) that represents an agent's trajectory.

   • Nodes ($V$): Represent distinct agent actions (e.g., view, str_replace, submit). Each node contains metadata including the action type, logical phase, navigation level, and outcome.
   • Temporal Edges ($TE$): Connect actions in chronological order.
   • Structural Edges ($SE$): Connect actions based on "subsuming" relationships within the problem space (e.g., a directory $\to$ file $\to$ code block). This captures navigation depth and breadth.
   • Phases: Nodes are labeled with logical phases: Localization (finding the bug), Patching (fixing the code), Validation (testing), and General.

2. Langutory: A human-readable abstraction of Graphectory.

   • It projects nodes onto a sequence of symbols representing logical phases (e.g., $L, P, V$).
   • It uses run-length encoding to compress consecutive identical phases (e.g., $L_5P_5LPV$), providing a compact summary of the agent's strategy.

B. Process-Centric Metrics & Analyses

The framework defines quantitative metrics and qualitative analyses derived from Graphectory:

• Metrics: Node Count (diversity of actions), Temporal Edge Count (trajectory length), Loop Count (repetitions/backtracking), Average Loop Length, Structural Edge Count (exploration depth), and Navigation Breadth.
• Phase Flow Analysis: Examines phase transition sequences to identify strategic shortcuts (skipping validation) or backtracks (revisiting localization after patching).
• Pattern Detection: Automatically mines anti-patterns (inefficiencies) such as "Repeated View," "Zoom Out" (navigating back up the hierarchy), "Overly Deep Zoom," and "Edit Reversion."
• Shared Strategy Analysis: Uses Generalized Sequential Pattern (GSP) algorithms to find common strategic subsequences across different agents and problems.

C. Online Monitoring and Intervention

The system supports real-time construction of Graphectory and Langutory during execution.

• Heuristics: Monitors for inefficiency signals (e.g., large loops, misaligned structural/temporal edges, long phase stagnation).
• Intervention: Upon detecting issues, the system notifies the agent with diagnostic messages. For severe plan violations (e.g., submitting without validation), it can rollback the trajectory to the previous state, forcing the agent to retry with corrected guidance.

3. Key Contributions

1. Graphectory & Langutory: Novel structural representations for agentic trajectories that capture both temporal and semantic relationships, moving beyond linear logs.
2. Process-Centric Metrics: A suite of quantitative metrics to measure trajectory complexity, efficiency, and strategy adherence.
3. Automated Analysis Pipeline: A system capable of analyzing 4,000 trajectories automatically, identifying common strategies, and detecting inefficiency patterns without manual annotation.
4. Online Intervention Mechanism: A technique to detect and correct trajectory issues in real-time, significantly improving resolution rates.
5. Comprehensive Dataset: A public dataset of 4,000 trajectories from 8 $\langle$agent, model$\rangle$ pairs solving 500 SWE-bench Verified issues.

4. Experimental Results

The authors evaluated two agents (SWE-agent and OpenHands) using four backbone LLMs (DeepSeek-V3, DeepSeek-R1, Devstral-small, Claude Sonnet 4) on 500 SWE-bench Verified issues.

• Correlation with Success & Difficulty:

  • Unresolved issues consistently exhibit larger Graphectories with more back edges, repetitions, and inefficient patterns compared to resolved ones.
  • Task Difficulty: As human-rated difficulty increases, agents perform more extensive exploration (higher Structural Edge Count) and exhibit more strategy shifts.
  • Model Strength: Stronger models (e.g., Claude Sonnet 4) generate more complex Graphectories with broader exploration and more validation steps, correlating with higher success rates.

• Strategy Analysis:

  • Successful runs typically follow a coherent $L \to P \to V$ flow.
  • Unsuccessful runs often show chaotic transitions, such as skipping validation ($L \to P \to \text{submit}$) or oscillating between phases without progress.
  • Stronger models demonstrate richer, more adaptive strategies, including deeper context gathering before patching.

• Inefficiency Patterns:

  • Even successful runs contain inefficiencies (e.g., redundant views, failed edits).
  • Common anti-patterns include Repeated View (re-opening the same file), Zoom Out (navigating back up the directory tree), and StrNotFound (editing failures due to string mismatches).
  • Unresolved runs often accumulate multiple anti-patterns within a single trajectory.

• Online Monitoring Impact:

  • Applying online monitoring and intervention to problematic instances resulted in a 6.9% to 23.5% increase in resolution rates across models.
  • It reduced oscillatory behavior by over 90% and significantly shortened trajectory lengths with near-zero overhead (<10ms).
  • It effectively prevented agents from submitting patches without validation or getting stuck in infinite loops.

5. Significance

This work fundamentally shifts the evaluation paradigm for agentic systems from outcome-centric to process-centric.

• Diagnostic Power: It provides the first systematic, scalable method to understand why agents fail or succeed, revealing hidden inefficiencies that outcome metrics miss.
• Actionable Insights: The identified anti-patterns (e.g., string-based editing failures) offer concrete targets for improving agent tooling and prompting.
• Real-Time Control: The online intervention capability demonstrates that process-aware monitoring can actively improve agent performance, suggesting a path toward self-correcting agentic systems.
• Foundation for Future Research: By providing a structured graph representation (Graphectory) and a large-scale dataset, the paper enables future research into symbolic navigation, AST-based editing, and more robust agent architectures.