🤖 machine learning

Guarantees on Dynamical System Distinguishability for LLM Token Generation

This paper provides a theoretical foundation for distinguishing LLM token generations by modeling them as dynamical systems, proving that classification accuracy improves exponentially with sequence length based on spectral distance while establishing fundamental limits and generalization bounds for cross-embedding transfer.

Original authors: Mohamed Akrout, Dan Wilson

Published 2026-08-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohamed Akrout, Dan Wilson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to tell two different storytellers apart. One tells the truth, and the other makes things up (hallucinates). If you only look at the individual words they use, you might be stuck. Both might use the same common words like "the," "cat," or "sat" with similar frequency. It's like trying to distinguish two different songs by only listening to the average volume of the notes; if the volume is the same, the songs sound identical. This is the problem with many current methods for checking if an AI is lying: they treat every word as an independent, random event, ignoring the fact that words come in a specific order and flow into one another.

However, if you listen to the rhythm and the melody—how the story moves from one sentence to the next—you might hear a pattern. In the world of science, this is called a "dynamical system." Think of it like a ball rolling down a hill. The ball's position at any single moment might look random, but the path it takes over time follows strict rules of physics. Large Language Models (LLMs) are like these rolling balls; they generate text by following hidden rules. A recent paper by Mohamed Akrout and Dan Wilson from the University of Tennessee asks a big question: Can we detect if an AI is hallucinating not by looking at the words it says, but by watching the invisible "dance" of how those words evolve? They treat the AI's internal thought process as a black-box machine and try to figure out if the machine is running on "truth mode" or "make-believe mode" just by watching its trajectory.

The paper proves that looking at the words alone (the "marginal distribution") is often a dead end. The authors show that two completely different machines—one telling the truth and one lying—can produce a mix of words that looks statistically identical. It's like two different chefs using the exact same ingredients in a bowl; if you just taste the soup, you can't tell who cooked it. The paper argues that any method ignoring the flow of the story is fundamentally limited and will hit a wall where it can't do better than random guessing.

But here is the exciting part: when you start looking at the movement, the story changes. The authors demonstrate that if you watch how the AI's internal state evolves over a sequence of words, the differences become huge. They prove mathematically that the chance of misclassifying the AI drops exponentially as the story gets longer. It's like listening to a song for just one second; you might not know the genre. But listen for thirty seconds, and the rhythm makes it obvious. The paper shows that the "rhythm" of a hallucinated story is fundamentally different from a factual one, and this difference grows stronger the more words you analyze.

Furthermore, the team tackles a tricky real-world problem: what if you train your detector on one type of AI "language" and try to use it on a different AI? Usually, this fails because the internal codes are different. However, the paper finds that if the two AI systems share a similar underlying geometric structure (like two different maps of the same city), the detector can transfer across them. They show that the success of this transfer depends on how well the two systems can be mathematically "translated" into each other. If the translation is clean, the detector works; if the translation is messy, the detector fails.

In short, this paper provides a rigorous mathematical guarantee that to catch an AI in a lie, you shouldn't just count the words; you need to watch the dance. By modeling the AI's output as a moving system rather than a static list of words, we can distinguish truth from fiction with high confidence, provided we give the system enough time to show its true rhythm. This shifts the focus from static snapshots of data to the dynamic patterns that emerge as the AI thinks, offering a new, more powerful way to understand and verify these complex machines.

Technical Summary: Guarantees on Dynamical System Distinguishability for LLM Token Generation

Problem Statement

Recent empirical work has demonstrated that Large Language Model (LLM) responses, specifically distinguishing between factual and hallucinated text, can be classified by modeling token embeddings as trajectories of a black-box dynamical system (DS). While this approach shows promise, the theoretical underpinnings remain unexplored. Specifically, three fundamental questions lack rigorous answers:

Necessity of Dynamics: If the marginal distributions of token embeddings differ between response categories, why is a dynamical model necessary?
Scalability and Performance: How does classification error scale with sequence length, and what specific dynamical quantities govern performance?
Generalizability: To what extent do token embeddings and their associated dynamics transfer across different embedding models, given that Koopman operator approximations are typically sensitive to the specific observables used?

Methodology

The authors formalize the classification task as a binary hypothesis test between two stochastic linear dynamical systems.

1. System Formulation

The authors model the evolution of lifted token embeddings $z_k$ (obtained via Extended Dynamic Mode Decomposition, EDMD) under two hypotheses:

$H_c$ (Correct/Factual): $z_{k+1} = A_c z_k + \xi^{(c)}_k$
$H_h$ (Hallucinated): $z_{k+1} = A_h z_k + \xi^{(h)}_k$

Here, $A_c$ and $A_h$ are finite-dimensional approximations of the Koopman operator for the respective regimes. The noise terms $\xi$ capture both approximation errors and genuine stochasticity, assumed to be i.i.d. Gaussian with distinct means and covariances.

2. Classification Mechanism

Classification is performed by comparing prediction residuals. For a given trajectory, the cumulative squared errors under both models ( $S_c$ and $S_h$ ) are computed. The decision rule relies on the differential residual score $\Delta E = \sqrt{S_h} - \sqrt{S_c}$ . A trajectory is classified as hallucinated if $\Delta E$ falls below a threshold $\eta$ .

3. Theoretical Framework

The paper derives bounds based on:

Stationary Distributions: Analyzing the marginal distributions of the observables.
Dynamical Discriminability ( $\delta^2$ ): A metric quantifying the expected squared model-mismatch signal per time step, defined as $\delta^2 = E[\|P\Delta z_k\|^2]$ , where $\Delta = A_c - A_h$ .
Intertwining Condition: A linear map $T$ relating two different embedding spaces, used to analyze cross-embedding transfer.

Key Contributions and Results

Result 1: Limitation of Marginal Density Analysis

The authors prove that two stable dynamical systems can possess completely different dynamics ( $A_c \neq A_h$ ) yet produce stationary marginal distributions that are nearly indistinguishable (Total Variation distance $\approx 0$ ).

Implication: Any classifier relying solely on static features (e.g., SVD modes or marginal statistics) faces a fundamental accuracy floor. If the marginal distributions overlap significantly, such classifiers cannot perform better than random guessing, regardless of the underlying dynamical differences. This explains empirical observations where static features of factual and hallucinated embeddings often overlap.

Result 2: Exponential Separability via Dynamics

The paper proves that a classifier based on prediction residuals achieves a misclassification probability that decays exponentially with the sequence length $L$ .

Governing Quantity: The decay rate is governed by the dynamical discriminability $\delta^2$ and the noise level.
Sample Complexity: To achieve a target error probability $\epsilon$ , the required sequence length scales as $L^* = O(\log(1/\epsilon))$ .
Significance: This result breaks the accuracy ceiling imposed by marginal analysis. It explains why richer embedding models (which likely yield higher $\delta^2$ ) can achieve high accuracy with shorter sequences, while simpler models require longer contexts.

Result 3: Cross-Embedding Transfer Bounds

The authors establish conditions under which dynamical models fitted on one embedding space can transfer to another without retraining.

Approximate Intertwining: Transfer is possible if the two embedding spaces share a geometric structure described by an affine transformation $G^{(\theta_2)}_2(q) \approx T G^{(\theta_1)}_2(q) + r$ .
Lower Bound: The transferable discriminability $\delta^2_{\theta_1 \to \theta_2}$ is lower-bounded by the smallest singular value of the transformation matrix $T$ and the original discriminability, penalized by the "intertwining defect" (reconstruction error $\chi^2$ ).
Implication: This explains why cross-embedding classification can perform above chance even without retraining, provided the geometric distortion between the embedding spaces is not too severe. It provides a theoretical basis for the surprising empirical finding that discriminative information survives drastic changes in embedding backends.

Significance and Conclusion

The paper provides a rigorous mathematical framework for understanding LLM response classification through the lens of dynamical systems theory. Its primary significance lies in shifting the analytical focus from static token representations to dynamical patterns emerging across token trajectories.

The work demonstrates that:

Dynamics are essential: Static marginal analysis is insufficient for distinguishing regimes with similar stationary distributions but different evolutions.
Temporal accumulation is powerful: Leveraging temporal dependencies allows for exponential error reduction as sequence length increases.
Transferability is structured: Cross-embedding generalization is not accidental but governed by the geometric compatibility (intertwining) of the embedding spaces.

By formalizing these guarantees, the paper motivates further investigation into using dynamical systems theory to analyze AI systems, contrasting with the more common paradigm of using AI to model dynamical systems. The authors note limitations, such as the reliance on linear Koopman approximations and Gaussian noise assumptions, suggesting future work could extend these bounds to nonlinear models and heavy-tailed distributions.