Critical Percolation as a Synthetic Data Model for Interpretability

Technical Summary: Critical Percolation as a Synthetic Data Model for Interpretability

Problem Statement
Deep neural networks learn features that reflect the hierarchical, multi-scale structure of natural data. However, synthetic datasets currently used to evaluate mechanistic interpretability methods often lack this realistic structure, limiting their utility as toy models. Existing synthetic models may capture sparsity or hierarchy in isolation but fail to unify properties such as power-law statistics, low intrinsic dimensionality, and self-similarity found in natural data distributions. Furthermore, the "curse of dimensionality" implies that natural data must possess high structure to be efficiently learnable. Current interpretability research, particularly regarding Sparse Autoencoders (SAEs), observes phenomena like feature splitting and absorption, suggesting that learned features possess a complex hierarchical organization that simple sparse models do not fully capture. There is a need for a principled, analytically tractable data model that incorporates these structural properties to serve as a realistic testbed for interpretability research.

Methodology
The authors introduce a family of synthetic datasets based on critical mean-field percolation theory. The model is constructed through the following components:

1. Data Generation via Percolation:

   • Space: Data points are drawn from a high-dimensional hypercubic lattice ($\Lambda \subset \mathbb{R}^d$).
   • Criticality: Sites are occupied with probability $p = p_c$ (the critical threshold), ensuring the system is in a critical regime where clusters exhibit fractal properties and power-law size distributions ($n_s \propto s^{-5/2}$).
   • Mean-Field Assumption: The dimension $d$ is assumed to be sufficiently large ($d \gg 6$) such that the lattice behaves like a Bethe lattice (a tree), making the model analytically tractable.
   • Geometry: The resulting clusters are sparse, low-dimensional fractals with intrinsic dimension $D=4$.

2. Hierarchical Target Generation:

   • Targets are generated by latent variables organized in a binary tree structure (a "latent forest").
   • As bonds between adjacent occupied sites are activated, they merge clusters. A new latent variable is introduced for each merge, acting as the parent to the latents of the merged sub-clusters.
   • The target value $y_x$ for a data point $x$ is a linear function of the latent variables along the path from the leaf (the point) to the root of its cluster's tree, normalized by depth.

3. The Cyclic Coalescent Algorithm:

   • Directly simulating percolation on high-dimensional lattices is computationally expensive and does not explicitly yield the hierarchical decomposition.
   • The authors leverage a mathematical equivalence between critical mean-field percolation clusters, uniform random labeled trees, and the additive coalescent process.
   • They propose a cyclic coalescent algorithm that jointly samples a random tree and its hierarchical latent decomposition. By arranging nodes in a random cyclic order and iteratively merging a node with its successor, the algorithm generates a uniform random tree and the corresponding hierarchy in almost linear time, $O(n \alpha(n))$, where $\alpha$ is the inverse Ackermann function.

4. Experimental Setup:

   • Two datasets were generated: a "one-cluster" dataset ($2 \times 10^5$ points) and a "multi-cluster" dataset ($2 \times 10^6$ points, filtered for cluster size $\ge 500$).
   • A residual Multi-Layer Perceptron (MLP), designed to resemble a transformer feed-forward block, was trained on these datasets.
   • Probing Experiments: Linear probes were trained to decode the ground-truth latent variables from the neural network's activations (input, hidden layers, and residual stream).

Key Contributions

1. A Principled Synthetic Data Model: The paper introduces a data model that unifies sparsity, taxonomic hierarchy, power-law statistics, low intrinsic dimension, and self-similarity, all derived from critical percolation theory.
2. Analytical Tractability: The model relies on known critical exponents (e.g., $\tau=5/2$, $D=4$) and does not require hyperparameter tuning to fix its statistical properties.
3. Efficient Sampling Algorithm: The authors develop and prove the correctness of the cyclic coalescent algorithm, enabling the generation of synthetic hierarchical percolation datasets at arbitrary scales in near-linear time.
4. Empirical Validation: The study demonstrates that ground-truth latent variables in this model can be linearly decoded from neural network activations, validating the model's utility for interpretability research.

Results

• Model Performance: The trained MLP achieved performance comparable to a 1-nearest-neighbor (1NN) baseline and significantly outperformed Ridge regression, confirming the task's nonlinearity and the model's learnability.
• Linear Decodability: Probing experiments revealed that the ground-truth latent variables are linearly representable in the network's activations.
  • Probes trained on hidden layer activations consistently achieved lower Mean Squared Error (MSE) than those trained on raw inputs.
  • The probe performance gap relative to raw input decreased as the "Figure of Merit" (FOM)—defined as the ratio of latent size to the square root of cluster size—increased.
• Power-Law Trends: The per-latent MSE exhibited a power-law trend as a function of FOM, suggesting that the statistical importance of a latent (determined by its size and cluster context) is a better predictor of decodability than its depth in the hierarchy alone.
• Layerwise Structure: No clear layerwise progression was observed; probe MSE was similar across all MLP blocks. This suggests the network does not necessarily build a hierarchical sum layer-by-layer in an interpretable manner, or that the hierarchy is represented differently than in compositional models.

Significance and Claims
The paper claims that critical percolation provides a principled testbed for interpretability research. By offering a dataset with known ground-truth hierarchical latents and realistic structural properties (sparsity, power laws, fractals), it allows researchers to:

• Validate and improve interpretability tools (like SAEs) against a known ground truth.
• Investigate whether SAE pathologies (feature splitting, absorption) arise when trained on data with taxonomic hierarchies.
• Study the relationship between data structure and the structure of neural activations.

The authors position this work as a complement to empirical studies of Large Language Models (LLMs), offering a controlled environment to test hypotheses about how neural networks learn hierarchical features. They explicitly note limitations, such as the model's assumption of random data population and its current support for scalar regression rather than tokenized classification, framing these as directions for future work rather than immediate solutions. The work does not claim to solve interpretability but to provide a rigorous, mathematically grounded synthetic environment to facilitate such research.