Alternative routes to universal diversity scaling in component systems: from proteomes to large language models

Technical Summary: Alternative Routes to Universal Diversity Scaling in Component Systems

Problem Statement
Complex "component systems"—ranging from genomes and proteomes to languages, LEGO constructions, and large language models (LLMs)—exhibit remarkably consistent statistical laws regarding diversity. Two primary regularities are observed:

1. Heaps' Law: The number of distinct components (vocabulary size, $h$) grows sublinearly with the total system size ($m$), typically as $\langle h \rangle \propto m^\nu$ with $0 < \nu < 1$.
2. Quadratic Fluctuation Scaling: The variance of the vocabulary scales quadratically with its mean ($\sigma_h^2 \propto \langle h \rangle^2$), indicating non-self-averaging behavior. This contrasts with simple sampling models (Poisson scaling, $\sigma_h^2 \propto \langle h \rangle$).

While these patterns are often interpreted as signatures of an underlying "innovation" mechanism (where new components are discovered as the system grows), the minimal ingredients required to generate both laws simultaneously remain poorly understood. Furthermore, it is unclear whether these statistical laws uniquely identify the generative mechanism or if alternative frameworks (e.g., latent heterogeneity) can produce identical macroscopic signatures.

Methodology
The authors employ a mathematical modeling approach to compare two broad explanatory frameworks:

1. Sequential Growth Models (Innovation-Duplication): These models assume a system grows by adding tokens via either duplication of existing components or the discovery of new ones (innovation). The authors derive analytical conditions for a general class of these models, specifically investigating the innovation probability $p_n(h, m)$. They analyze the asymptotic behavior of the mean and variance using continuous-time approximations and stability analysis of the mean-field trajectory.
2. Sampling Models with Latent Heterogeneity: These models assume realizations are generated by sampling from a component frequency distribution (often power-law/Zipfian) that varies across realizations due to "latent variables" (e.g., topics in texts, taxonomic groups in genomes). The authors utilize the law of total variance to decompose the total variance into within-topic and between-topic components.

The study validates these theoretical predictions against empirical data from four distinct datasets:

• Project Gutenberg Books & Wikipedia: Text corpora analyzed for word diversity.
• LEGO Constructions: Sets analyzed for brick type diversity, annotated by themes.
• Proteomes: Bacterial genomes analyzed for protein domain family diversity, annotated by taxonomic clades.
• Large Language Models (LLMs): Text generated by GPT-2 and Llama-3 models using random prompts.

Key Contributions and Results

1. Necessary Conditions for Innovation Models
The authors derive that for a general innovation-duplication model to reproduce both Heaps' law and quadratic fluctuation scaling, the innovation probability must satisfy a specific asymptotic dependence:
$$p_n \sim \frac{h}{m}$$
This represents a "diversity-generates-diversity" mechanism, where the probability of discovering a new component increases linearly with the current diversity ($h$) and decreases with system size ($m$).

• If $p_n$ depends only on $m$ (e.g., standard Yule-Simon models), the system can reproduce sublinear growth but is bounded by Poisson scaling ($\sigma_h^2 \leq \langle h \rangle$), failing to capture the observed quadratic fluctuations.
• The authors demonstrate that this linear dependence is a necessary condition within this class of models; other functional forms fail to jointly reproduce the empirical scaling exponents.

2. Emergence of Scaling in Large Language Models
The study tests the "diversity-generates-diversity" hypothesis in LLMs. By analyzing the output vocabulary of GPT-2 and Llama-3 given random prompts, the authors find that the effective innovation probability is indeed linear in the current vocabulary size ($p_n \propto h/m$). This suggests that despite the complexity of LLMs, their generative process effectively implements the specific innovation rule required to produce the observed statistical laws.

3. Latent Heterogeneity as an Alternative Route
The authors demonstrate that quadratic fluctuation scaling arises naturally in sampling models with latent variables (e.g., topics) without requiring any explicit innovation mechanism.

• Using the law of total variance, they show that if the average vocabulary size depends on a latent variable (e.g., $\langle h \rangle_t = k(t)m^\nu$), the variance across realizations includes a term proportional to the variance of the latent variable: $\sigma_h^2 \approx \langle h \rangle + \text{CV}^2[k(t)] \langle h \rangle^2$.
• For sufficiently large system sizes, the quadratic term dominates, reproducing the empirical scaling.
• Crucially, in this framework, the "effective" innovation probability inferred from the data ($p_n \approx d\langle h \rangle/dm$) also converges to the linear form $p_n \sim h/m$, not because of an explicit rule, but as a statistical consequence of the latent heterogeneity.

4. Empirical Validation and Indistinguishability
Analysis of genomes (taxonomic groups) and LEGO sets (themes) shows that conditioning on known annotations separates realizations into groups with distinct diversity levels (different prefactors $k(t)$) while preserving power-law growth within groups. However, residual quadratic fluctuations persist even after conditioning, suggesting that known annotations capture only part of the latent structure.

• Conclusion on Mechanism: The paper concludes that the two frameworks—innovation-driven growth and latent-variable sampling—are phenomenologically indistinguishable based solely on Heaps' law, fluctuation scaling, and inferred innovation probabilities. Both routes lead to the same macroscopic statistical laws.

Significance and Claims
The paper claims to provide a clearer interpretation of diversity scaling in component systems by revealing a conceptual duality:

• No Unique Identification: The observed statistical laws (Heaps' law and quadratic fluctuations) do not uniquely identify the underlying generative mechanism. They can arise from an explicit "rich-get-richer" innovation process or from the statistical aggregation of heterogeneous latent states.
• Unified Statistical Picture: The authors propose a unified view where latent structure accounts for component frequencies, average diversity, and diversity fluctuations. The "diversity-generates-diversity" mechanism observed in growth models may be an emergent effective description of underlying latent heterogeneity.
• Cautionary Note: The findings serve as a warning against interpreting diversity scaling as definitive evidence of a specific innovation process. In systems like LEGO constructions or gene expression profiles, where sequential innovation is not the natural description, latent variables provide a more parsimonious explanation.
• Future Directions: The paper identifies the need for observables beyond second-order statistics (e.g., higher-order correlations) or systems with fully characterized latent variables to discriminate between these mechanisms.

The work bridges complex systems theory, statistical physics, and biology, suggesting that the universality of diversity scaling may stem from general statistical principles (like the law of total variance) rather than specific dynamical rules.