Technical Summary: LEDGER – Claim-to-Evidence Trace Graphs for Auditing LLM Agents

Problem Statement

As Large Language Model (LLM) agents evolve from single-turn question-answering systems into interactive workers capable of executing long-horizon technical workflows (involving tool use, code execution, file edits, and artifact generation), the primary bottleneck in productivity has shifted from output generation to auditability. While existing observability systems (e.g., LangSmith) provide fine-grained visibility into execution events (prompts, tool calls, errors, intermediate outputs), this visibility does not equate to auditability.

Reviewers currently face a "flat record" problem: they must manually reconstruct the logical structure of a session to determine which actions, artifacts, and validation steps support a specific conclusion. This manual reconstruction is labor-intensive, scales poorly with session complexity, and obscures the direct lineage between a final claim and its underlying evidence. The paper argues that effective auditing requires an evidence-centered approach where reviewers can traverse from a reported result back to the concrete actions and artifacts that produced it, rather than reading a linear log.

Methodology: LEDGER System

The authors introduce LEDGER (Layered Evidence and Decision Graphs for Execution Review), a sidecar tracing and review system that operates alongside unmodified interactive agent sessions. LEDGER does not replace observability but reorganizes captured records into a layered semantic trace graph designed for human review.

1. Capture and Trace Records

The foundation of LEDGER is the Trace Record, a stable, non-interpretive substrate of captured session data.

• Mechanism: The system uses lifecycle hooks (e.g., SessionStart, PreToolUse, PostToolUse) and transcript reconstruction to capture JSON payloads containing messages, tool invocations, results, and file interactions.
• Integrity: These records preserve the original order and content of the session, including links to the source transcript. They serve as the "source of truth," distinct from any inferred structure.

2. Layered Graph Construction

LEDGER organizes Trace Records into a three-tiered graph structure:

• Evidence Nodes: These group closely related Trace Records (e.g., a tool call and its result) into inspectable work units. They are categorized by Type (Action vs. Artifact) and Category (e.g., user_message, tool_call, control, artifact). Artifact nodes specifically represent inspectable objects like code patches, plots, tables, or command outputs.
• Workflow Nodes: These group related Evidence Nodes into higher-level task phases (e.g., context, plan, inspect, execute, validate, claim). This abstraction allows reviewers to view the session at a phase level rather than an event-by-event level.
• Semantic Edges: Typed, directed edges connect nodes to define relationships. Key edge types include:
  • uses: A work unit consumes an artifact.
  • produces: A work unit creates or modifies an artifact.
  • checked_by: A change is validated by a specific step.
  • supports: Evidence justifies a claim.
  • informs: A result shapes a subsequent plan.
  • frames: A requirement sets the context for a task.

3. Interface and Review Workflow

The system provides a local dashboard that integrates:

• Graph Views: A two-level visualization (Workflow and Evidence layers) allowing reviewers to traverse from high-level phases down to specific evidence.
• Artifact Inspection: Direct indexing into underlying artifacts (e.g., opening a specific plot, patch, or table) linked from the graph.
• Trace-Construction Audit: Views showing the raw Trace Records and graph updates, enabling reviewers to distinguish between agent errors and tracing errors (i.e., verifying how the graph was constructed from the raw data).

Key Contributions

1. Claim-to-Evidence Trace Graph Construction: A method for parsing unmodified agent sessions into a layered semantic graph where artifact nodes represent inspectable evidence and typed edges explicitly connect claims to supporting actions and validation steps.
2. Evidence-Centered Review Interface: A dashboard that unifies graph-level audit paths with source records and artifacts, allowing reviewers to move fluidly between the logical structure of the workflow and the raw evidence.
3. Separation of Capture and Interpretation: A design that strictly separates deterministic source records (Trace Records) from the inferred structure (Evidence/Workflow Nodes), ensuring that the graph remains an audit aid rather than an opaque source of truth.

Results and Case Studies

The paper validates LEDGER through two case studies using a Codex agent with live tracing enabled:

• Case Study 1: Tabular Data Analysis: An agent analyzed air quality data to generate a daily pattern report. The trace graph successfully exposed the artifact lineage, linking the final claim back through generated plots and summary tables to the source data cleaning steps. It also highlighted an error-and-repair sequence, showing how a failed script execution (due to a missing dependency) was traced, patched, and re-validated, making the repair process transparent.
• Case Study 2: Feature Addition in Codebase: An agent added a shortest-path utility to the NetworkX library. The graph distinguished between the initial implementation and subsequent regression testing and guard patches. It allowed reviewers to trace the design choice (placing the function in a specific module) back to repository inspection and documentation reads, and forward to the specific tests that validated the behavior.

In both cases, the system demonstrated the ability to make the "audit path" explicit, allowing reviewers to verify not just that a claim was made, but how it was supported by specific artifacts and checks.

Significance and Claims

The paper positions LEDGER as a necessary evolution in agent observability. Its significance lies in shifting the paradigm from visibility (seeing what happened) to auditability (understanding why a conclusion is trustworthy).

• Modest Claims: The authors explicitly state that the graph construction is not fully deterministic; the tracer interprets which records belong together and assigns semantic edges. Therefore, the graph is presented as an audit aid, not a source of truth. The interface is designed to keep underlying records visible so reviewers can verify the graph's construction.
• Future Direction: The paper suggests that future work should aim to replace model-inferred structure with deterministic or independently verifiable structure (e.g., via stronger instrumentation or provenance-aware backends) and improve the visual vocabulary to better distinguish between deterministic and inferred relations.

Ultimately, LEDGER aims to support the growing need for human oversight in complex agent workflows by preserving access to source records while making the connections between actions, artifacts, and conclusions inspectable and traversable.