🤖 machine learning

Approximate Homomorphisms and Convergent Representations in Transducers

This paper establishes theoretical conditions for the stability and convergence of minimal representations in controlled stochastic processes under perturbations, demonstrating that while standard transducers may lack structural robustness, finite-rank linear and predictive transducers exhibit approximate homomorphisms that support the hypothesis of structural convergence in neural network latent representations.

Original authors: Santiago Cifuentes

Published 2026-08-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Santiago Cifuentes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, humming landscape of modern artificial intelligence, a curious pattern has emerged. When researchers train different neural networks—complex systems designed to learn from data—to solve similar problems, they often find that the internal workings of these machines begin to look surprisingly alike. Even if the networks are built with different architectures or trained on slightly different datasets, the hidden layers where they store their partial calculations seem to converge on a shared structure. This observation has led to a bold idea: that these artificial minds are not just memorizing answers, but are independently discovering a common, underlying model of how the world works. To understand if this convergence is a fundamental law of learning or just a coincidence, scientists need a way to measure the "shape" of these internal models. They need to know if two different machines are truly thinking in the same way, or if they are just arriving at similar answers through completely different, incompatible paths.

This question sits at the heart of a new study that treats these internal models as mathematical machines called transducers. Think of a transducer as a device that takes a sequence of actions, like a series of commands or observations, and produces a sequence of reactions, like a series of outputs or predictions. It is a way to formalize how an agent, whether a robot or a neural network, interacts with its environment. The researchers wanted to see if these machines, when they are slightly different due to the noise of training or different data, still share a core, minimal structure. They asked: if two machines behave almost the same, can we map one onto the other in a way that preserves their logic, even if the mapping isn't perfect?

The team began by testing this idea on the most general type of these machines, which they call standard transducers. They hoped to find that any two machines behaving similarly could be simplified down to a single, shared blueprint. However, their investigation revealed a surprising limitation. They proved that for these general machines, there are specific behaviors where no such shared blueprint exists. Even if two machines are almost identical in their output, their internal structures can be so fundamentally different that they cannot be mapped onto one another without breaking the logic of how they work. This finding rules out the possibility that all convergent structures in artificial intelligence are simple or universal; it shows that without specific constraints, the path to a shared internal model can be blocked.

The story changes, however, when the researchers look at two more specific types of machines: linear transducers and predictive transducers. Linear transducers are those where the internal state is represented as a point in a geometric space, a structure that naturally fits the way modern neural networks operate, since their parameters are essentially numbers in a vector space. Predictive transducers are those where the internal state contains only the information strictly necessary to predict the future, discarding any redundant history. For both of these specific types, the researchers found a robust form of convergence, but with important caveats. For linear transducers, they demonstrated that this convergence holds specifically for interfaces of finite rank; if you take any two minimal machines of this type that implement interfaces of finite rank and behave almost the same, you can always find a single, minimal machine that acts as a common ancestor. You can map the complex, noisy versions of these machines onto this simple, clean version, and the errors introduced by the mapping will remain small and proportional to how different the original machines were. Similarly, for predictive transducers, they showed that convergence occurs when the machines are close according to a specific "residual metric" that compares their behavior after every possible history, regardless of how likely that history is.

This result is significant because it provides a theoretical foundation for the observation that neural networks often develop similar internal representations. It suggests that while general, unstructured models might not converge, the specific kinds of structures that modern AI tends to use—those that rely on linear geometry with finite-rank interfaces or efficient prediction under specific metrics—do have a natural tendency to settle into a shared, minimal form. The study confirms that for these specific architectures, the "Platonic Representation Hypothesis," the idea that different models converge on a shared statistical reality, holds true under the condition of small perturbations and specific structural constraints. The researchers showed that this convergence is not a fragile coincidence but a stable property of the mathematics governing these systems under these defined conditions.

The work also clarifies the conditions under which this stability breaks down. The researchers identified that the convergence for linear transducers relies on the interfaces being of finite rank, while for predictive transducers, it depends on the machines being close in the residual metric. When these conditions are met, the system is robust; when they are not, the internal structures can drift apart wildly. This distinction helps explain why some models might share deep structural similarities while others do not. It suggests that the success of finding shared representations in AI is not guaranteed for every possible architecture, but is a feature of those that adhere to certain mathematical constraints, particularly finite-rank linearity and predictive efficiency under the right metrics.

Ultimately, this study offers a map for understanding the hidden layers of artificial intelligence. It tells us that the search for a shared "language" of thought among different AI models is a valid pursuit, but only if we look at the right kind of models. The convergence is real, but it is not universal; it is a property of systems that are built to be stable and efficient within specific mathematical bounds. By proving that these specific types of machines have a unique, minimal core that all similar versions can be reduced to, the researchers have provided a rigorous explanation for why different AI systems often end up thinking in the same way. They have shown that the universe of possible machine behaviors contains islands of stability where different paths inevitably lead to the same destination, offering a glimpse into the mathematical laws that govern the emergence of intelligence in silicon.

Technical Summary: Approximate Homomorphisms and Convergent Representations in Transducers

Problem Statement

The paper investigates the theoretical stability of minimal representations of controlled stochastic processes (specifically transducers) under perturbations. This inquiry is motivated by empirical observations in deep learning where different neural network architectures, trained on different data or objectives, often exhibit convergent latent structures. This phenomenon has led to the "Platonic Representation Hypothesis," suggesting that agents learning from similar distributions converge to a shared statistical model of reality.

While previous work [44] established that minimal implementations exist for linear and predictive transducers (providing a unique "canonical" representation), these results were exact. In practical scenarios, models learn from slightly different datasets and are subject to noise, meaning they implement interfaces that are approximately similar rather than identical. The central problem addressed is whether the existence of a unique minimal representation (and thus convergent structure) is robust to these perturbations. Specifically, if two transducers implement interfaces that are $\epsilon$ -close, do they share a common structural representation (an approximate homomorphism) that maps them to a canonical form with error scaling linearly with $\epsilon$ ?

Methodology

The author formalizes the problem using three classes of transducers:

Standard Transducers: General controlled stochastic systems with hidden states.
Linear Transducers: Systems where states are embedded in a vector space, utilizing Hankel matrices for construction.
Predictive Transducers: Systems where internal states contain no predictive information unavailable from the observable history (equivalent to $\epsilon$ -machines in computational mechanics).

The methodology proceeds in three stages:

Definition of Approximate Homomorphisms: The author introduces $\epsilon$ -homomorphisms, which relax the exact structural mapping conditions of standard homomorphisms. For standard transducers, this involves bounding the Total Variation (TV) distance between the pushforward of the source dynamics and the target dynamics. For linear transducers, it involves bounding the norm of the difference between the mapped dynamics and the target dynamics.
Metric Spaces for Interfaces: To measure the similarity of the behaviors (interfaces) implemented by transducers, the author defines metrics over the space of interfaces. They utilize the supremum metric ( $d_\infty$ ) and discounted metrics ( $d_\gamma$ ) which weight long-horizon discrepancies less heavily. For predictive transducers, a "residual metric" ( $d_{res}$ ) is introduced, which compares the conditional interfaces of all admissible histories, regardless of their probability.
Stability Analysis: The author analyzes the set of transducers implementing interfaces within an $\epsilon$ -neighborhood of a target interface $I$ . They investigate whether this set possesses a " $\delta$ -minimum"—a transducer $T_{min}$ such that all other transducers in the set admit an approximate reduction to $T_{min}$ with error $\delta = O(\epsilon)$ .

Key Contributions and Results

1. Approximate Homomorphisms and Composability

The paper defines $\epsilon$ -homomorphisms for all three transducer types. A key theoretical property proven is composability: if $T_1 \xrightarrow{\epsilon_1} T_2$ and $T_2 \xrightarrow{\epsilon_2} T_3$ , then $T_1 \xrightarrow{\epsilon_1 + \epsilon_2} T_3$ . Furthermore, the author proves that approximate reductions preserve the induced interface dynamics under discounted metrics. Specifically, if $T_1$ reduces to $T_2$ with error $\epsilon$ , the distance between their interfaces under a discounted metric $d_\gamma$ is bounded by $O(\epsilon)$ .

2. Negative Result for Standard Transducers

For standard transducers, the author demonstrates that convergence fails even under approximation. They construct a specific interface $I$ such that for any $\delta < 1$ , the set of transducers implementing $I$ (or interfaces arbitrarily close to it) does not possess a $\delta$ -minimum. This implies that for general stochastic systems, the lack of a unique minimal representation cannot be salvaged by introducing an error term; structurally distinct implementations of similar dynamics may not share a common approximate structure.

3. Positive Result for Linear Transducers

For linear transducers, the paper proves a strong stability result. For any finite-rank interface $I$ , there exists a neighborhood of interfaces (within distance $\epsilon$ ) such that all minimal linear transducers implementing these nearby interfaces admit an approximate reduction to the canonical minimal implementation of $I$ .

Mechanism: The proof relies on the construction of the minimal linear transducer via the Hankel matrix. Small perturbations in the interface lead to small perturbations in the Hankel matrix. The author constructs a linear map (based on invertible minors of the Hankel matrix) that maps the perturbed canonical representation to the original one.
Error Bound: The error of this reduction scales linearly with the perturbation size ( $\delta \approx \Gamma_I \epsilon$ ), where $\Gamma_I$ is a constant depending only on the interface $I$ .

4. Positive Result for Predictive Transducers

For predictive transducers, the author shows that convergence holds under a specific metric. They introduce the residual metric ( $d_{res}$ ), which compares the conditional interfaces of all histories.

Condition: If the distance between two interfaces $I$ and $J$ under $d_{res}$ is sufficiently small (specifically, smaller than half the minimum separation $\Delta_I$ between distinct predictive states), then the minimal predictive transducer for $J$ admits an $\epsilon$ -reduction to the minimal predictive transducer for $I$ .
Significance: This result highlights that while the supremum metric fails to guarantee stability for predictive transducers (due to rare but distinct histories), a metric that accounts for the structure of all conditional histories restores stability.

Significance and Claims

The paper claims to provide theoretical support for the Platonic Representation Hypothesis under specific structural constraints.

Robustness of Abstractions: The results identify conditions under which canonical transducer representations are robust to perturbations. Specifically, if world models in AI architectures can be abstracted as linear or predictive transducers, then models trained on similar data distributions should exhibit structural convergence (i.e., their latent representations can be mapped to a common minimal form with bounded error).
Limitations of Generality: The paper explicitly states that this convergence is not guaranteed for general (standard) transducers, suggesting that the specific algebraic or structural properties of linear and predictive systems are necessary for the "convergent phenomenon" to emerge.
Theoretical vs. Empirical: The author frames these results as theoretical evidence. They note that while the existence of predictive structures (like $\epsilon$ -machines) in neural networks has been observed empirically [46, 47], the presence of linear transducers is a hypothesis based on the vector-space nature of neural parameters. The paper does not claim to have experimentally verified these structures in existing models but rather provides the mathematical framework to expect them.

In summary, the work establishes that the "convergence" of internal representations is a property that depends heavily on the class of mathematical abstraction used to model the system. It holds robustly for linear and predictive systems but fails for general stochastic systems, offering a nuanced theoretical explanation for why convergent structures might appear in some AI architectures but not others.