Scale-Dependent Semantic Dynamics Revealed by Allan Deviation

By applying Allan deviation to sentence embeddings, this study characterizes the semantic dynamics of text as a stochastic trajectory, revealing distinct short-time scaling regimes that differentiate creative from technical writing and exposing a systematic reduction in long-term stability horizon in large language models compared to human cognition.

The Gist

Imagine you are walking through a giant library. As you move from one book to the next, the "vibe" or meaning of the story changes. Sometimes the story jumps wildly to a new idea; other times, it stays very steady and predictable.

This paper treats the flow of a story (or any text) like a physical journey. Instead of just reading the words, the researchers measured how much the "meaning" of the text wiggles and shifts as you read sentence by sentence. They used a special mathematical tool called Allan deviation (usually used to measure the stability of atomic clocks) to see how stable the story's direction is over time.

Here is what they found, explained simply:

1. The "Wiggle" Test: Creative vs. Technical

The researchers looked at two types of books: Creative stories (like novels and poems) and Technical manuals (like science textbooks or encyclopedias).

• Creative Stories are like a Jazz Improvisation: When you read a novel, the meaning between sentences jumps around a bit more freely. It's less predictable. The math showed that the "wiggles" in meaning are wilder and less connected to the previous sentence.
• Technical Texts are like a Train on Tracks: In a science textbook, every sentence must follow logically from the one before it. If you change the order, the meaning breaks. The math showed that these texts have very tight, strong connections between sentences. The meaning doesn't wander; it stays on a strict path.

The Analogy: Imagine walking through a park.

• In a novel, you might suddenly decide to run to a tree, then sit on a bench, then chase a butterfly. Your path is full of small, unpredictable turns (high "wiggle").
• In a textbook, you are walking a straight line. You can't turn left or right without breaking the logic of the path. Your steps are very uniform (low "wiggle").

2. The "Stamina" Test: How Long Can the Story Go?

The researchers also asked: How long can a text keep this interesting, shifting pattern before it gets stuck or repetitive? They called this the "Context Horizon."

• Human Writers (The Marathon Runners): When humans write stories, they can keep the "wiggles" and interesting shifts going for a very long time. Even after hundreds of sentences, the story finds new ways to explore ideas without getting stuck in a loop.
• AI Models (The Sprinters): The researchers tested large language models (AI chatbots). At the very beginning (sentence 1 to 10), the AI sounds just like a human. The "wiggles" look the same. However, as the text gets longer, the AI runs out of steam much faster.
  • Around sentence 18 to 26, the AI's text starts to flatten out. It stops finding new ideas and starts repeating the same semantic "noise."
  • Humans, by contrast, kept their interesting patterns going until sentence 37 or much longer.

The Analogy: Imagine two people telling a story to a group.

• The Human can keep the story fresh and surprising for a long time, constantly introducing new characters or twists.
• The AI starts great, but after a while, it starts circling back to the same few ideas. It runs out of "new ground" to cover and gets stuck in a rut, even though it started sounding perfect.

3. Why This Matters

The paper claims that this method proves meaning has a physical "shape" that we can measure.

• It shows that human creativity has a specific "fingerprint" of freedom and long-term stability.
• It reveals that current AI, while very good at mimicking humans in the short term, lacks the deep, long-term ability to keep exploring new semantic territory without getting stuck.

In a nutshell: The paper uses a clock-making tool to measure how "steady" or "wild" a story is. It found that human stories are wilder and last longer before getting repetitive, while AI stories are more predictable and run out of fresh ideas much sooner.

Technical Summary

Technical Summary: Scale-Dependent Semantic Dynamics Revealed by Allan Deviation

Problem Statement
While language is recognized as a complex emergent system with established properties like Zipf's law and long-range correlations, the specific dynamics of how meaning evolves sequentially through a text remain elusive. Previous studies utilizing random-walk mappings, power spectra, and detrended fluctuation analysis have successfully identified scale-free structures but fail to explicitly probe temporal ordering. Consequently, these methods cannot directly characterize the dynamics of semantic progression or identify characteristic scales where qualitative changes in semantic organization occur. The central question addressed is whether the evolution of a narrative through a semantic state space resembles a stochastic random walk or exhibits long-range correlations analogous to critical or driven physical systems.

Methodology
The authors propose a framework treating the semantic progression of written text as a stochastic trajectory in a high-dimensional state space, analyzed through the lens of precision metrology.

1. Signal Construction: Text corpora are segmented into ordered sentences, $S(t)$, which are mapped to fixed-dimensional embedding vectors, $v_t$, using transformer-based sentence-embedding models (primarily all-MiniLM-L6-v2).
2. Semantic Displacement: The instantaneous semantic distance, $d_t$, is calculated as the cosine distance between consecutive sentence embeddings ($v_t$ and $v_{t+1}$).
3. Cumulative Signal: These incremental displacements are accumulated to form a one-dimensional stochastic signal, the "cumulative semantic phase," $\phi(t) = \sum_{i \le t} d_i$. This representation preserves temporal ordering while discarding absolute semantic coordinates, focusing the analysis on the dynamics of change rather than content.
4. Allan Deviation Analysis: The authors apply Allan deviation, $\sigma_y(\tau)$, a variance estimator typically used to distinguish short-time variability from long-range drift in sequential data. Here, the averaging scale $\tau$ represents the number of sentences used for coarse-graining.
   • Short-time regime: Probes sentence-to-sentence variability.
   • Long-time regime: Reveals the accumulation of correlations over extended text stretches.
   • Scaling Law: The analysis examines the power-law behavior $\sigma(\tau) \sim \tau^\alpha$, where the exponent $\alpha$ reflects the correlation structure of the underlying increments.

Key Contributions

• Application of Allan Deviation to Semantics: The paper introduces Allan deviation as a novel tool for quantifying semantic dynamics, offering a method to separate short-time fluctuations from long-range drift in text.
• Definition of "Context Horizon": The authors define a physically interpretable "semantic stability horizon" (or context horizon) as the crossover point where the Allan deviation curve flattens into a noise floor. This marks the scale beyond which additional averaging fails to reduce semantic variance, indicating a loss of effective semantic convergence.
• Genre Differentiation: The study establishes that short-time scaling exponents serve as quantitative fingerprints that distinguish creative literature from technical and informational texts.
• Human vs. Model Comparison: The framework provides a metric to differentiate human cognitive dynamics from algorithmic generation by analyzing the stability horizon of semantic progression.

Results

1. Genre-Dependent Scaling Exponents:

   • Creative Texts: Novels, drama, epic poetry, and short stories exhibit steeper short-time scaling exponents (averaging $\alpha \approx -0.39$ to $-0.40$), clustering near the white-noise limit ($\alpha = -0.5$). This indicates weak local correlations and greater freedom in semantic exploration between successive sentences.
   • Technical Texts: Physics, biology, chemistry, mathematics, and encyclopedic texts display significantly shallower slopes (averaging $\alpha \approx -0.25$ to $-0.30$). This reflects stronger local correlations and more constrained semantic evolution required for conceptual consistency.
   • Validation: Randomizing sentence order within texts eliminates these distinctions, confirming that the observed scaling arises from ordered semantic structure rather than embedding geometry or text length.

2. Context Horizon Variations:

   • Technical and informational texts exhibit crossovers to a noise floor at relatively small fractions of their total length (e.g., Biology at 4.125, Chemistry at 5.34, normalized units).
   • Creative texts sustain power-law scaling over substantially larger scales. Notably, novels did not exhibit a comparable crossover within the measured range, suggesting scale-invariant semantic organization.

3. Human vs. Large Language Models (LLMs):

   • Local Similarity: At short averaging scales, model-generated text (from models including GPT-4, Claude 3, Llama 3, etc.) exhibits power-law scaling exponents statistically indistinguishable from human-written text.
   • Stability Horizon Divergence: At larger scales, model-generated text deviates from the short-time regime significantly earlier than human text.
     • Human Text: Maintains stable scaling up to a context horizon of approximately 37 sentences.
     • LLM Text: Exhibits reduced stability horizons, ranging from 13 to 26 sentences depending on the model.
   • Interpretation: This earlier crossover suggests that autoregressive models converge to a core idea quickly, running out of semantically distinct concepts and failing to explore the semantic space for new connections as effectively as human authors.

Significance and Claims
The paper claims to establish semantic coherence as a measurable physical property. By treating semantic progression as a dynamically varying temporal signal, the authors provide a framework to:

• Differentiate the nuanced dynamics of human cognition from patterns generated by algorithmic models.
• Quantify the "freedom of semantic exploration" inherent in creative literature versus the "constrained semantic space" of factual writing.
• Offer a bridge between statistical physics and language analysis that is agnostic to specific linguistic content, embedding choices, or model architectures.

The authors modestly position this work as a method to probe the stability of meaning and the dynamics of semantic drift, rather than a tool for content generation or specific NLP tasks. The results suggest that while LLMs successfully mimic local scaling statistics, they exhibit a systematic reduction in their long-range semantic stability horizon.