💻 computer science

DMDIntel: Interpreting Large Language Models via Dynamic Mode Decomposition

This paper introduces DMDIntel, a novel interpretability framework that leverages dynamic mode decomposition to decompose LLM hidden states into prominent modes and rank input tokens, demonstrating superior performance over state-of-the-art attribution methods like PCA, integrated gradients, and SHAP across multiple datasets and model families.

Original authors: Amogh Joshi, Animesh Mukherjee, Sergey Utyuzhnikov

Published 2026-08-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Amogh Joshi, Animesh Mukherjee, Sergey Utyuzhnikov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a magician pull a rabbit out of a hat. You see the hat, you see the rabbit, but the magic happens in the split second between the two, hidden inside the magician's sleeves. For a long time, Artificial Intelligence (AI) has been that magician. We give it a sentence, and it gives us an answer, but nobody really knows how it decided that answer. The AI is a "black box," and scientists are desperate to peek inside to see which words the computer actually cared about. This field is called "interpretability." To understand the new paper, you first need to know two things: how AI reads text and how we usually try to peek inside.

AI models, specifically the ones called "Large Language Models" (LLMs), don't read words like we do. Instead, they turn every word into a long list of numbers (a vector) and pass them through a series of layers, like a relay race. At each layer, the numbers change slightly, mixing the meaning of the current word with the words that came before it. By the end of the race, the model has a final set of numbers that decide the answer. Usually, to figure out which words mattered, scientists use tools that look at the "static" state of the model—like taking a single photo of the relay race and guessing who was running the fastest. But this paper suggests that looking at a single photo misses the whole story. It argues that we should watch the movie of the race, looking at how the numbers evolve from one step to the next, because the secret to the AI's thinking is in that movement.

Enter DMDINTEL, a new method introduced by Amogh Joshi and his team from IIT Kharagpur and the University of Manchester. Think of the AI's internal processing not as a static list of numbers, but as a flowing river. As the AI reads a sentence, the "water" (the hidden information) ripples and changes shape with every new word. The researchers used a mathematical tool called Dynamic Mode Decomposition (DMD)—originally invented to study how water swirls in fluid mechanics—to map these ripples.

Here is how their "river" analogy works in practice: When the AI reads a sentence, the hidden states of the words create a complex, swirling pattern. DMDINTEL breaks this pattern down into a few simple, repeating "modes" or shapes, much like how a complex wave in the ocean can be described as a combination of a few basic wave shapes. The team found that by watching how these shapes grow, shrink, or oscillate as the AI reads the sentence, they could identify which words were the "drivers" of the final decision. They treated the AI as a dynamical system, meaning they looked at the change between one word and the next, rather than just the word itself.

The researchers tested this idea on three different families of AI models (Llama, Qwen, and Mistral) and three different tasks: figuring out if a review is positive or negative, spotting fake news, and detecting hateful speech. They compared their "river movie" method against the old "static photo" methods, which include techniques like Integrated Gradients, SHAP, and Principal Component Analysis (PCA).

The results suggest that DMDINTEL is a much better detective. In their experiments, the new method consistently found the most important words (the "ground truth") more often than the other methods. For example, when the AI was deciding if a review was negative, DMDINTEL correctly ranked words like "trash" and "unable" as the top drivers, whereas other methods sometimes got distracted by less important words like "scheduled" or "conference." The paper shows that by looking at the evolution of the AI's thoughts, DMDINTEL captures the logic of the sentence better. It suggests that the AI's reasoning isn't just about the words sitting still; it's about how the meaning flows and transforms as the sentence is built.

However, the authors are careful to note that this isn't a magic bullet for every AI problem. Their method works best when the AI is just reading a sentence and giving a simple answer (like a classification task). They admit that if the AI starts writing a long story or translating a whole book word-by-word, the math gets tricky because the AI starts feeding its own new words back into the system. But for the specific job of explaining why an AI labeled a sentence as "hate speech" or "fake news," this new way of watching the numbers dance suggests a clearer, more accurate picture of the machine's mind.

Technical Summary: DMDINTEL – Interpreting LLMs via Dynamic Mode Decomposition

1. Problem Statement

The paper addresses the challenge of interpretability in large language models (LLMs), specifically regarding input attribution in classification tasks. While traditional methods like Integrated Gradients (IG), SHAP, and Principal Component Analysis (PCA) are effective at identifying which tokens a model attends to, the authors argue they have significant limitations:

Discrete Treatment: They often treat tokens as discrete, isolated entities.
Lack of Temporal Context: They fail to capture the sequential evolution of latent information as the model processes a sequence token-by-token.
Fragmented Snapshots: They provide a static view of reasoning rather than a continuous narrative of how hidden states evolve to form a prediction.

Existing mechanistic interpretability work often focuses on static probing or specific circuit mapping but struggles to model the dynamic flow of information across a sequence.

2. Methodology: The DMDINTEL Framework

The proposed framework, DMDINTEL, treats a supervised fine-tuned decoder-only LLM as a discrete-time dynamical system. It models the evolution of hidden states as a flow in the embedding space, approximating the non-linear model function $F$ with a linear surrogate operator.

Core Workflow

Dynamical System Formulation:
The model's state transition is modeled as $h_{t+1} \approx F(w_{\le t})$ , where $h_t$ is the hidden state at token $t$ . The authors approximate this transition using a linear relationship $h_{t+1} = A h_t$ , where $A$ is a surrogate linear operator.
Data Matrix Construction & Denoising:
- Layer Selection: A heuristic (Algorithm 1) selects the optimal decoder layer ( $\ell^*$ ) by analyzing the cosine similarity between token hidden states and an "instruction bias" vector (derived from an empty prompt). The selected layer balances input-token fidelity against instruction-following bias, typically located at the inflection point where similarity to the instruction bias begins to rise sharply.
- Bias Removal: To isolate the signal of the actual input sentence from the prompt template, the authors subtract the instruction bias vector ( $b$ ) from the hidden states: $\hat{h}_i = h_i - b$ .
- Matrix Formation: The debiased hidden states form a data matrix $X$ .
Dynamic Mode Decomposition (DMD):
- Standard DMD: The linear operator $A$ is approximated from the data matrix. Its eigendecomposition yields DMD modes (spatio-temporal structures) and eigenvalues (quantifying growth, decay, and oscillation).
- Higher-Order DMD (HODMD): To capture long-term dependencies and temporal dynamics that standard DMD (which assumes a Markovian $h_{t+1} = A h_t$ ) might miss, the authors also employ HODMD. This constructs a Hankel matrix from stacked snapshots, allowing the system to model hidden states as linear combinations of previous time steps.
Input Attribution (Token Ranking):
- Mode Ranking: Modes are ranked by their dynamical significance, specifically using the magnitude of initial modal amplitudes or the time-averaged magnitude of amplitudes.
- Token Scoring: For each token $w_t$ , the contribution is measured by the state change $\Delta h_t = h_t - h_{t-1}$ . The scalar projection of this change onto the top- $k$ dominant modes is calculated. The sum of these projection magnitudes across the top modes yields a projection score ( $s_t$ ) for the token.
- Final Output: Tokens are ranked by their projection scores to identify the most influential drivers of the model's prediction.

3. Key Contributions

DMDINTEL Framework: Introduction of an input attribution pipeline that decomposes LLM hidden states into interpretable spatio-temporal modes to rank input tokens.
Dynamical Perspective: The first work to analyze the sequential evolution of hidden states in decoder-only LLMs as a dynamical system specifically for input attribution.
Empirical Validation: Extensive evaluation across three model families (Llama-3.2-3B, Qwen3-4B, Mistral-7B) and three diverse text classification datasets (Sentiment, HateXplain, FakeEdit).
Performance Superiority: Demonstration that DMDINTEL consistently outperforms state-of-the-art baselines (PCA, IG, SHAP) in identifying ground-truth tokens while preserving their ranked importance.

4. Experimental Results

The authors evaluated the framework using Matched Count (MC), Rank-Biased Overlap (RBO), and Recall@k against ground truth (GT) rankings generated by GPT-4.1.

Sentiment Analysis: HODMD with time-averaged amplitude ranking performed best. The authors attribute this to the need for delay factors to capture multi-word phrase dynamics, which standard DMD misses. DMDINTEL outperformed PCA, IG, and SHAP across all metrics.
HateXplain & FakeEdit: Standard DMD with amplitude-based ranking proved most effective. The authors suggest that for these tasks, where specific keywords (e.g., hate speech terms or political figures) drive the classification, the state evolution follows a near-Markovian process that does not require the extended memory of HODMD. DMDINTEL achieved the highest Recall@10, retrieving the most ground-truth tokens.
Qualitative Analysis: DMDINTEL showed strong alignment with GT, consistently ranking semantically relevant tokens (e.g., "trash," "unable," "fa**ot") near the top. In contrast, IG often focused on narrow subsets of toxic keywords while missing context, and SHAP sometimes surfaced contextually plausible but GT-absent tokens.
Fidelity (Perturbation): In token-masking experiments, DMDINTEL consistently caused significant drops in model accuracy and confidence (often ranking first or second), confirming that the identified tokens are indeed critical to the model's decision-making process.
Robustness: The method showed stability across different model families and was relatively insensitive to small variations in the selected layer index ( $\ell^*$ ).

5. Significance and Claims

The paper claims that DMDINTEL offers a more robust and effective lens for input attribution compared to traditional static or gradient-based methods. By viewing the LLM as a dynamical system, the framework captures the continuous narrative of model reasoning rather than fragmented snapshots.

The authors emphasize that:

The hidden states of fine-tuned models, when decomposed into lower-dimensional spatio-temporal modes, reveal interpretable patterns that serve as robust indicators of information flow.
This dynamical perspective allows for the identification of tokens that drive the output classification with higher fidelity, particularly in tasks where sequential context or specific token-level triggers are critical.
The approach successfully bridges the gap between complex non-linear model behavior and interpretable linear structures (modes) without requiring prior knowledge of the system's governing equations.

6. Limitations

The authors acknowledge specific boundaries to their current work:

Task Scope: The framework is currently evaluated only on classification tasks with short outputs (1-2 tokens). It is noted that applying this to auto-regressive generation (e.g., machine translation) is challenging because generated tokens are fed back into the model, complicating the formulation of the DMD matrix.
Model Scope: The study focuses on supervised instruction fine-tuned models for specific downstream tasks. General-purpose instruction-tuned models released by companies remain unexplored in this context.