Technical Summary: LLM Explainability with Counterfactual Chains and Causal Graphs

Problem Statement

Large Language Models (LLMs) exhibit broad capabilities, yet their inference mechanisms remain opaque. Decision factors are unobservable, and generated explanations often lack faithfulness, posing risks in high-stakes domains like medicine and law. While existing interpretability methods (e.g., attention analysis, feature attribution) offer correlational insights, they fail to provide the causal evidence required for faithful explanations. Furthermore, mechanistic interpretability focuses on low-level components (neurons, circuits) that often misalign with the high-level concepts stakeholders need to understand.

Current causal approaches in LLMs typically model real-world structures (e.g., how changing an author's gender affects a prediction) or assume a fixed causal graph (concepts $\to$ text $\to$ prediction). There is a critical gap in methods that recover the LLM-inference causal graph itself: a structure that maps how a specific model organizes human-interpretable concepts to reduce input text to a prediction (text $\to$ concepts $\to$ prediction). A major bottleneck in this recovery is data coverage; reliable causal discovery requires dense sampling of concept combinations, which is often impossible or unethical in real-world settings.

Methodology

The authors propose a fully automated, model-driven four-phase pipeline to construct concept-level causal graphs for classification tasks. The method uses the target LLM itself to extract concepts, represent data, and generate the counterfactuals necessary for robust causal discovery.

1. Label Prediction

The ground-truth labels in the dataset are replaced with the target LLM's own predictions ($\hat{y}$). This ensures the resulting graph reflects the model's internal reasoning rather than the dataset's ground truth.

2. Discriminative Concept Discovery and Annotation

The framework iteratively identifies a set of human-interpretable concepts ($C$) that differentiate between the predicted classes.

• Extraction: The LLM proposes candidate concepts based on batches of text and predicted labels.
• Filtering: Concepts are retained only if they are both relevant (present in the text) and discriminative (aligned with a strict subset of classes, not all or none).
• Annotation: Each input text is mapped to a concept vector $\phi(x)$, where each concept $c_i$ is assigned a categorical value representing the subset of classes the model perceives it to support.

3. MCMC-Inspired Data Expansion

To address the sparsity of observational data in the latent concept space, the authors introduce a counterfactual augmentation procedure inspired by Markov Chain Monte Carlo (MCMC).

• Process: Starting from an original text, the algorithm iteratively proposes textual perturbations to shift the alignment of specific concepts toward or away from target classes.
• Acceptance Criteria: A proposed counterfactual is accepted only if it achieves the intended conceptual shift (target alignment) and minimizes unintended changes to non-target concepts (minimal side effects).
• Refinement: Failed proposals undergo recursive refinement, where the LLM is prompted to correct specific violations.
• Outcome: This generates a denser dataset ($D_{mcmc}$) that covers diverse combinations of concept states, enabling stable causal discovery.

4. Causal Discovery via $\sigma$-CG

The expanded, annotated dataset is fed into the $\sigma$-CG algorithm, a causal discovery method capable of handling discrete variables and potentially cyclic structures. The resulting directed graph $G$ captures the causal flow from input text, through LLM-perceived concepts, to the final prediction, with constraints ensuring text is a root and prediction is a sink.

Key Contributions

1. Causal Paradigm for Global Explainability: The paper introduces a framework that maps LLM reasoning to causal graphs over human-interpretable concepts, moving beyond local, input-centered explanations to a global account of the inference mechanism.
2. MCMC-Inspired Data Augmentation: A novel algorithm that generates counterfactuals to densely populate the latent concept space, overcoming the data coverage bottleneck inherent in causal discovery for black-box models.
3. Evaluation Protocol: A rigorous protocol for assessing structural stability and predictive fidelity in the absence of ground-truth reasoning graphs, demonstrating that discovered causal parents are superior predictors of nodes compared to alternative concept subsets.

Results

The framework was evaluated on three LLMs (Gemini-2-Flash, Qwen3-14B, gpt-OSS-20B) across three tasks: Disease Diagnosis (synthetic), Sentiment Analysis (IMDB), and LLM-as-a-Judge (Reddit).

• Predictive Fidelity: In all settings, the causal graph's parent sets outperformed random concept subsets in predicting both the final label and intermediate concept states. In most cases, the graph-based parents ranked in the top 3 predictors, confirming that the discovered structures capture meaningful dependencies.
• Model-Specific Reasoning: On synthetic tasks, different models converged on similar explanatory concepts (recovering ground-truth variables). However, on naturalistic data, models developed distinct latent heuristics and causal topologies, highlighting that model selection requires understanding not just performance but also reasoning alignment.
• Convergence and Stability: The MCMC expansion demonstrated distributional convergence, with Kullback-Leibler (KL) divergence stabilizing as chains approached the theoretical lower bound of perfect overlap. Structural stability was confirmed as the Hamming distance between successive graph topologies dropped to zero.
• Impact of Augmentation: Counterfactual expansion significantly improved predictive accuracy over original data alone, proving the necessity of the augmentation stage for robust causal discovery.

Significance and Claims

The paper positions this work as a foundational step toward "concept-level explainability" for LLMs. By modeling the inference process as a causal graph over human-interpretable concepts, the authors argue that this representation offers a more accessible and actionable form of interpretability for both developers and domain experts.

The authors modestly claim that this work provides a foundation for a new generation of process-oriented explainability methods. They emphasize that their approach addresses the specific challenge of data sparsity in latent spaces through their MCMC-inspired expansion, enabling the recovery of global causal structures that were previously inaccessible. The work does not claim to solve all interpretability issues but establishes a viable pipeline for discovering how models organize concepts to make decisions, distinguishing between models that may perform similarly but reason differently.