From Actions to Understanding: Conformal Interpretability of Temporal Concepts in LLM Agents

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed as autonomous agents capable of multi-step reasoning and planning in interactive environments (e.g., ScienceWorld, AlfWorld). However, their internal decision-making processes remain "black boxes."

• The Gap: Traditional interpretability methods (e.g., feature visualization, static probing) focus on single inputs or static embeddings. They fail to capture the temporal dynamics of an agent's reasoning.
• The Challenge: An agent might start a task correctly but drift into failure modes (hallucinations, irrelevant exploration) midway through a trajectory. Existing methods cannot pinpoint when or where in the sequence the internal representation diverges from a "success" state to a "failure" state, as they often rely only on the final binary outcome (success/failure) of the entire episode.
• Goal: To develop a framework that interprets the temporal evolution of concepts within LLM agents, specifically identifying linearly separable directions in the model's activation space that correspond to "success" vs. "failure" at every individual time step.

2. Methodology

The authors propose a Step-wise Conformal Interpretability Framework consisting of three core components:

A. Step-Wise Reward Modeling

Instead of relying solely on the sparse final reward of an episode, the authors generate dense, step-wise feedback signals.

• Technique: They utilize Monte Carlo sampling over future trajectories (following Xiong et al.).
• Process: Given the current state and history $\tau_t$, the agent samples $N$ complete future trajectories based on its learned policy.
• Output: A step-wise reward $r_t$ is calculated as the average expected future success probability conditioned on the current step. This transforms the problem from a sparse final signal to a dense temporal signal indicating the likelihood of success at each specific step.

B. Statistical Labeling via Conformal Prediction

To rigorously label these step-wise rewards as "success" or "failure" without arbitrary thresholds, the authors employ Inductive Conformal Prediction (ICP).

• Non-Conformity Measure (NCM): They define NCMs based on the reward values: $\alpha_s = 1 - r_t$ (for success) and $\alpha_f = r_t$ (for failure).
• Calibration: Using a calibration set, they compute p-values ($p_s$ and $p_f$) for each step.
• Labeling Logic: A step is labeled "success" if $p_s \geq \epsilon_s$ and $p_f < \epsilon_f$ (and vice versa for failure).
• Theoretical Guarantee: This approach provides provable error bounds. The probability of mislabeling a true success as failure (False Negative) is strictly bounded by $\epsilon_s$, and similarly for False Positives by $\epsilon_f$.

C. Probing Latent Directions

Once steps are statistically labeled, the authors investigate the geometry of the model's internal representations.

• Hypothesis: Directions of success and failure are linearly separable in the LLM's activation space across different layers and time steps.
• Implementation: They train linear probes (logistic regression classifiers) on the residual stream activations ($h^L_t$) of the LLM at specific layers $L$ and time steps $t$.
• Objective: The probes attempt to classify the hidden state as "success" or "failure" based on the conformally labeled ground truth. High probe accuracy validates the existence of distinct, linearly separable directions for these temporal concepts.

3. Key Contributions

1. Temporal Interpretability Framework: A novel method to decompose the "black box" of an LLM agent's trajectory into step-wise, statistically calibrated success/failure signals.
2. Conformal Labeling: The first application of Conformal Prediction to assign bounded-error labels to temporal representations in LLM agents, moving beyond heuristic thresholds.
3. Discovery of Linear Separability: Empirical evidence showing that LLMs fine-tuned for sequential tasks develop internal, linearly separable representations for success and failure across time steps, layers, and domains (in-distribution and out-of-distribution).
4. Steering Mechanism: A demonstration that these identified "success directions" can be used to steer the agent during inference, correcting reasoning drift and hallucinations in real-time.

4. Experimental Results

The framework was evaluated on Llama-2-7B agents in two environments: ScienceWorld (scientific reasoning) and AlfWorld (household tasks).

• Linear Separability:

  • ScienceWorld: Linear probes achieved extremely high accuracy (often 100%) and F1 scores across most layers and time steps for both in-distribution (ID) and out-of-distribution (OOD) tasks.
  • AlfWorld: Probes showed strong performance (Accuracy 60–95%, F1 0.56–0.95), though slightly lower than ScienceWorld. The authors attribute this to ScienceWorld's denser reward structure (sub-goals) compared to AlfWorld's single final reward.
  • Conclusion: Success and failure concepts are indeed linearly separable in the latent space, even for OOD tasks.

• Steering Performance:

  • The authors applied Representation Engineering (RepE) to steer the agent toward the identified "success direction" at an early intervention point (Step 3).
  • Result: The steered model showed a 1.1% accuracy boost over the baseline. While modest, this improvement was achieved with a single, low-cost intervention, demonstrating the practical utility of the framework for early failure detection and correction.
  • Qualitative Improvement: Steering successfully mitigated common failure modes, such as hallucinating object possession or looping due to incorrect sub-goal sequencing.

5. Significance and Future Impact

• Trustworthy Autonomy: This work provides a principled method for early failure detection. Instead of waiting for an agent to fail at the end of a task, practitioners can monitor internal representations to detect drift mid-trajectory.
• Intervention Capability: By identifying the geometric direction of "success," the framework enables targeted interventions (steering) to guide agents back on track without retraining the model.
• Foundation for Safety: The use of Conformal Prediction ensures that the interpretation signals come with statistical guarantees, a critical requirement for deploying autonomous agents in safety-critical domains.
• Future Directions: The authors plan to extend this framework to multimodal embodied settings and develop more sophisticated, proactive steering strategies for temporal data.

In summary, the paper bridges the gap between static interpretability and dynamic agent behavior, proving that LLM agents possess internal, linearly separable concepts of success and failure that can be statistically identified and utilized to improve performance.