Phase transition in large language models and the criticality of natural languages

Technical Summary: Phase Transition in Large Language Models and the Criticality of Natural Languages

Problem Statement
Natural languages exhibit universal statistical features, most notably power-law behaviors such as Zipf's law (word frequency vs. rank) and Heaps' law (vocabulary size vs. document length). In statistical physics, such power-law behavior is a hallmark of systems at a critical point, situated near a phase transition between distinct macroscopic phases. This observation motivates the conjecture that the stochastic process generating natural language lies on a critical surface separating a "repetitive" phase from an "incomprehensible" phase. However, testing this hypothesis is fundamentally difficult because real-world natural languages lack controllable parameters; one cannot vary a "temperature" or external field to observe a transition. Previous attempts using simplified mathematical models have provided limited insight into real-world linguistic phenomena.

Methodology
To overcome the lack of controllability in natural languages, the authors utilize Large Language Models (LLMs) as effective, controllable models. Specifically, they employ the 160M-parameter Pythia model trained on the Pile (a large-scale English dataset). The study treats the generation of text as a stochastic process where the temperature parameter $T$ (analogous to physical temperature in the Boltzmann distribution) is varied during sampling.

The authors map generated texts to sequences of Part-of-Speech (POS) tags (18 distinct values) to facilitate statistical analysis. They systematically analyze texts generated across a range of temperatures ($T$) and training steps ($k$) to investigate:

1. Correlation Decay: Measuring the time correlation $C(t, t+\Delta t)$ between POS tags to determine if correlations decay exponentially (disordered) or follow a power law (critical).
2. Integrated Correlation ($\tau$): Calculating the sum of correlations to distinguish between short-range and long-range order as a function of sequence length $N$.
3. Power Spectrum Analysis: Analyzing the power spectrum $S(\omega)$ of POS sequences to detect periodic structures and counting the number of peaks ($n_{peaks}$) to characterize repetitive patterns.
4. Training Dynamics: Repeating these analyses on partially trained models (at various training steps $k$) to determine when these critical features emerge.
5. Perplexity Evaluation: Using perplexity on the Pile dataset to quantitatively measure how closely the generated texts at different $T$ and $k$ match natural language statistics.

Key Results

1. Phase Transition at $T_c \approx 1$: The analysis reveals a distinct phase transition in the LLM at a critical temperature $T_c \approx 1$.

   • Low-Temperature Phase ($T < 1$): Texts exhibit long-range order with correlations converging to a positive value. The power spectrum shows a divergent number of peaks ($n_{peaks} \propto N$), indicating complex, inhomogeneous repetitive structures rather than simple periodicity.
   • High-Temperature Phase ($T > 1$): Texts are disordered and incomprehensible. Correlations decay rapidly (short-range), and the power spectrum contains only a single peak at $\omega=0$.
   • Critical Point ($T \approx 1$): Correlations follow a power-law decay ($C \sim \Delta t^{-\eta}$ with $\eta \approx 0.2$), and the integrated correlation $\tau$ diverges with sequence length $N$. This behavior mirrors the power-law statistics observed in natural languages.

2. Emergence During Training: These phenomena are not inherent to the model architecture but emerge as the model learns the natural language.

   • At early training steps ($k \lesssim 100$), the model generates disordered text regardless of temperature, and no phase transition is observed.
   • Critical behavior (divergent $\tau$) emerges around $k_c \approx 102$.
   • Complex repetitive structures (divergent $n_{peaks}$) emerge slightly later, around $k \approx 10^3$.
   • This indicates that both criticality and complex repetition are intrinsic properties acquired through training on natural language data.

3. Optimal Match to Natural Language: The study finds that the perplexity of the model on the natural language dataset is minimized at the final training step ($k=143,000$) and at the critical temperature $T \approx 1$. This quantitatively confirms that the stochastic process defined by the LLM at the critical point most closely resembles the statistical properties of natural language.

Significance and Claims
The paper argues that these findings provide strong evidence for the conjecture that natural languages are critical. The authors propose a schematic model where the stochastic process of natural language lies on a critical surface separating repetitive and incomprehensible phases. An LLM trained on this language defines a one-dimensional manifold in the space of stochastic processes; this manifold intersects the critical surface near the point corresponding to the natural language when the temperature is set to the critical value.

The significance of this work lies in:

• Empirical Validation: Moving beyond toy mathematical models to demonstrate phase transitions in realistic, large-scale language models.
• Statistical Physics Perspective: Providing a framework to characterize natural languages through the lens of critical phenomena, suggesting that the power-law behaviors observed in language are signatures of the system operating near a critical point.
• Universality: Preliminary results suggest these phenomena are robust across different model sizes and languages (Chinese, French, Japanese, Korean, Spanish), implying a universal critical nature of natural languages.

The authors conclude that natural languages are indeed critical, with the LLM serving as a controllable effective model that allows for the empirical observation of this criticality, which is otherwise inaccessible in real-world language data.