Absolute abstraction: a renormalisation group approach

This paper argues that true abstraction in neural networks depends not only on depth but also on the breadth of the training data, proposing a renormalisation group framework where expanding the data scope leads representations toward a unique "Hierarchical Feature Model," a hypothesis validated by experiments with Deep Belief Networks and auto-encoders.

The Gist

The Big Idea: How to Think Like a Genius

Imagine you are trying to learn about the world.

• Level 1 (The Details): You see a specific golden retriever named "Buster." You notice his floppy ears, his brown spots, and that he likes to chase tennis balls.
• Level 2 (The Category): You see a poodle, a beagle, and Buster. You realize they are all "dogs." You ignore the spots and the specific names.
• Level 3 (The Abstract): You see a dog, a cat, and a horse. You realize they are all "animals." You ignore the fact that one barks and one meows.
• Level 4 (Absolute Abstraction): You realize that everything in the universe is made of "matter" and "energy." You have stripped away so many details that you are left with a universal truth that applies to everything, not just dogs or cats.

This paper asks a simple question: How does a computer (or a brain) get from Level 1 to Level 4?

Most people think the answer is just "depth." If you stack enough layers of a neural network (like adding more floors to a building), it will eventually become abstract. The authors say: "No, that's not enough."

They argue that to reach Absolute Abstraction, you need two things working together:

1. Depth: Having many layers to process information.
2. Breadth: Seeing a massive variety of different things (not just dogs, but cats, cars, clouds, and galaxies).

The Analogy: The "Zoom Lens" of the Universe

The authors use a concept from physics called the Renormalization Group (RG). Let's translate that into a photography analogy.

Imagine you have a camera with a magical zoom lens.

• Zooming In (Details): You look at a single pixel on a photo of a beach. You see sand grains. This is "low-level" data.
• Zooming Out (Abstraction): You zoom out. The sand grains blur together. You see a beach. You zoom out more. You see an ocean. You zoom out even more. You see the Earth.

The paper suggests that to get a truly "universal" view (like seeing the Earth from space), you can't just zoom out on one photo of a beach. You have to zoom out on photos of every beach, every forest, every city, and every mountain.

If you only zoom out on one specific beach, you just get a blurry version of that beach. But if you zoom out on everything, the specific details (the color of the sand, the shape of the waves) disappear, and you are left with the fundamental rules of how the world is organized.

The "Hierarchical Feature Model" (The Perfect Summary)

The paper proposes that when you combine Deep Layers with Broad Data, the computer's internal brain settles into a specific state called the Hierarchical Feature Model (HFM).

Think of the HFM as the Ultimate Cheat Sheet.

• In a normal brain, if you learn about "cats," you remember "fur," "whiskers," and "meowing."
• In the HFM, the brain organizes information by how much detail is needed.
  • Some things need very few bits of info to describe (e.g., "It exists").
  • Some things need a lot of bits (e.g., "It is a specific type of cat with a specific scar").

The HFM is special because it is data-independent. It doesn't care if you are looking at a cat, a car, or a cloud. It only cares about the structure of the information. It's like a universal translator that speaks the language of "complexity" rather than the language of "cats."

The Experiments: Teaching Computers to See the Big Picture

The authors tested this theory using two types of AI:

1. Deep Belief Networks (DBNs): These are like deep stacks of filters.
2. Auto-Encoders: These are like compression algorithms that try to shrink a picture down to its essence and then rebuild it.

The Experiment:
They trained these AIs on pictures.

• Scenario A: They trained the AI only on the number "2" from the MNIST dataset (a standard set of handwritten numbers).
• Scenario B: They trained it on "2"s, then added "3"s, then "4"s, then letters, then fashion items, and finally pictures of cars (Cifar-10).

The Result:

• When the AI only saw "2"s, its internal representation was messy and specific to "2."
• As they added more types of data (Breadth) and more layers (Depth), the AI's internal "brain" started to look exactly like the Hierarchical Feature Model.
• The AI stopped caring about whether it was looking at a "2" or a "cat." It started organizing everything based on how "complex" or "detailed" the object was.

Why Does This Matter?

This paper suggests that Intelligence isn't just about memorizing facts. It's about finding the universal patterns that connect everything.

• For AI: If we want AI to be truly smart and adaptable, we shouldn't just make it deeper. We must feed it a wider, more diverse diet of data.
• For Humans: It explains why a child who grows up seeing many different animals, cultures, and ideas becomes better at abstract thinking than a child who only sees one type of thing.
• The "Platonic" Connection: The authors mention that all these different AIs, when trained on broad data, seem to converge on the same "statistical model of reality." It's as if they all discover the same "Universal Grammar" of the universe, regardless of what they were originally taught.

The Takeaway

To reach Absolute Abstraction, you need to zoom out on the widest possible universe.

If you only look at a small part of the world, you get stuck in the details. But if you look at everything through many layers of processing, the details fade away, and you are left with the pure, universal structure of reality. That is what the authors call "Absolute Abstraction."

Technical Summary

1. Problem Statement

The paper addresses the nature of abstraction in machine learning and cognitive systems. While it is well-established that deep neural networks (DNNs) develop increasingly abstract representations through hierarchical layering (combining low-level features like edges into high-level concepts like objects), the authors argue that depth alone is insufficient to achieve "truly" or "absolutely" abstract representations.

The core problem is defining and characterizing a representation that is universal and data-independent. The authors posit that true abstraction emerges only when the process of removing irrelevant details is iterated indefinitely on a dataset that simultaneously expands in breadth (variety of data domains). They seek a theoretical framework to identify the "fixed point" of this process—a representation that encodes the organization of data irrespective of the specific content.

2. Methodology

The authors employ a Renormalization Group (RG) framework, drawing an analogy between statistical physics and the learning process in neural networks.

Theoretical Framework: RG Transformation

The authors model the evolution of a representation $p(s)$ (a probability distribution over hidden binary variables) as it adapts to broader data domains. They define two complementary RG transformations:

1. Zooming Out ($\Re^\uparrow$): Expanding the data domain. This involves:
   • Coarse-graining: Integrating out fine-grained details (low-level features).
   • Rescaling: Introducing new, independent high-level features to account for the broader universe of data while maintaining a fixed coding cost (entropy).
   • Renormalization: Adjusting the distribution to preserve the information capacity (entropy) of the representation.
2. Zooming In ($\Re^\downarrow$): Focusing on a specific subclass of data while enriching it with finer details.

The Fixed Point:
The authors prove that both transformations converge to a unique fixed point distribution, denoted as $p^*$. This fixed point is identified as the Hierarchical Feature Model (HFM), previously introduced in [7].

• HFM Definition: A maximum entropy model where features are organized hierarchically. The probability of a state depends solely on $m_s$, the index of the most detailed active feature (the "level of detail").
• Key Property: The HFM satisfies the Principle of Maximal Relevance, meaning the coding cost (information content) is distributed broadly, allowing the representation to be maximally informative and data-independent.

Empirical Validation

To test the theoretical predictions, the authors conducted numerical experiments on two architectures trained on datasets of varying breadth:

1. Deep Belief Networks (DBNs): Stacked Restricted Boltzmann Machines (RBMs).
2. Auto-Encoders (AEs): Symmetric encoder-decoder networks.

Datasets:
They used variants of the MNIST dataset, progressively expanding the "breadth" of the training data:

• Single digit (e.g., only '2') $\to$ Transformed '2's $\to$ All MNIST digits $\to$ EMNIST letters $\to$ Fashion-MNIST $\to$ CIFAR-10.

Metric:
They measured the distance between the empirical internal representation of the networks and the theoretical HFM using the Kullback-Leibler (KL) divergence. To make this comparison valid, they optimized a transformation (permutation and bit-flipping) to align the network's hidden variables with the hierarchical structure of the HFM.

3. Key Contributions

1. Definition of Absolute Abstraction: The paper proposes that "absolute abstraction" is not a static property but the fixed point of a Renormalization Group transformation driven by the combined effects of increasing depth (hierarchical processing) and breadth (diversity of training data).
2. Universality of Representation: It provides a theoretical proof that under these conditions, the internal representation converges to a universal distribution (the HFM) that is independent of the specific data being represented. Data-specific information is relegated to the parameters connecting the hidden layers to the visible layer.
3. Connection to Maximal Relevance: The authors demonstrate that the HFM, as the fixed point, naturally satisfies the Principle of Maximal Relevance, linking information-theoretic efficiency with the emergence of abstract concepts.
4. Empirical Confirmation: The study provides the first numerical evidence that standard deep learning architectures (DBNs and AEs) spontaneously approach this theoretical fixed point when trained on increasingly broad datasets.

4. Results

• Convergence to HFM: In both DBNs and AEs, the KL divergence between the internal layer distributions and the HFM decreases as both the depth of the network and the breadth of the training data increase.
• Necessity of Breadth: For a fixed depth, increasing breadth initially improves the match to the HFM, but eventually, without sufficient depth, the representation diverges. Conversely, depth alone without breadth does not lead to the fixed point.
• Parameter Evolution: The parameter $g$ of the HFM (which controls the "temperature" or entropy of the distribution) decreases as data breadth increases, consistent with the theory that broader data requires a more "ordered" (lower entropy relative to the maximum) but more general representation.
• Robustness: The results hold across different training orders, dataset segmentation strategies, and training durations, provided the training is sufficient to reach equilibrium.
• Critical Point: The fitted parameter $g$ approaches the critical value $g_c = \log 2$, suggesting the system operates near a phase transition where the representation balances between random noise and rigid structure, a hallmark of efficient coding.

5. Significance

• Theoretical Unification: The paper bridges statistical physics (RG theory), information theory (maximal relevance), and machine learning (deep learning), offering a unified mathematical description of how abstraction emerges.
• Platonic Representation Hypothesis: The findings strongly support the hypothesis that neural networks trained on diverse data converge to a shared, universal statistical model of reality, independent of the specific task or modality.
• Understanding Intelligence: The authors suggest that the "universal grammar" or abstract reasoning capabilities of human intelligence may arise from a similar RG process in the brain, where the integration of vast, multi-modal sensory inputs drives the system toward a fixed-point representation.
• Learning vs. Fitting: The work distinguishes between fitting (estimating parameters to reproduce data) and learning (discovering the variation and structure of data). It suggests that true understanding involves mapping data onto a universal scaffold (the HFM), making the system robust to new, unseen tasks that share abstract commonalities.

In conclusion, the paper argues that absolute abstraction is a statistical inevitability for deep learning systems exposed to sufficiently broad and deep data, characterized mathematically by the Hierarchical Feature Model.