🤖 machine learning

Beyond Uniform Local Isometry and Topology: FactoMap for Disentangled Representations

This paper introduces FactoMap, a novel disentanglement method that learns interpretable prototypes on a factor-space lattice to capture complex geometric structures like wrapping, collapsing, and non-uniform scales, thereby enabling the separation of statistically independent factors that are not geometrically separable in standard Euclidean spaces.

Original authors: Sohini Gupta, Bahareh Tolooshams

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sohini Gupta, Bahareh Tolooshams

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quest to make artificial intelligence truly understand the world, researchers have long pursued a specific kind of clarity. Imagine trying to describe a complex scene, like a busy street, to a computer. If the computer simply memorizes the pixels, it cannot easily tell the difference between a red car and a blue car, or a large truck and a small one. To solve this, scientists developed a method called disentangled representation learning. The goal is to teach a machine to break down an image into its independent building blocks, or "factors," such as color, size, and position. In an ideal system, changing just the color of an object in the data should change only one specific number in the computer's internal code, leaving the numbers for size and position completely untouched. This separation is crucial because it allows machines to generalize better, make fairer decisions, and be controlled more precisely. However, a persistent assumption has guided this field: that these building blocks behave like a flat, uniform grid, where every step in any direction feels the same distance.

This assumption, however, does not always hold true in the real world. A new study from the University of Alberta challenges the idea that these factors can always be mapped onto a simple, flat surface. The researchers, Sohini Gupta and Bahareh Tolooshams, discovered that even when factors are statistically independent—meaning they are chosen without regard to one another—they can still be geometrically tangled. They demonstrated that the way a factor affects an image can change depending on the current state of other factors. For instance, changing the size of an object might alter how much its color appears to shift in the final image, creating a relationship where the "distance" between colors grows or shrinks as the object gets larger. This means that a fixed, uniform map cannot accurately represent the data. To solve this, the team introduced a new method called FactoMap, which builds a flexible, structured map that can stretch, wrap, and collapse to match the true shape of the data.

The core of the problem lies in how different factors interact with the process of rendering an image. In many previous approaches, scientists assumed that if you moved a certain amount in the "color" direction, it would always produce the same visual change, regardless of the object's size. The researchers showed that this is often false. When they simulated objects with varying hues and scales, they found that the visual impact of changing the hue depended entirely on the scale. As an object grew larger, the effect of changing its color became more pronounced relative to the effect of changing its size. This created a situation where the geometry of the data was not a simple rectangle or a flat sheet, but something more complex, where the "distance" between two colors was not constant but varied with the size of the object. Furthermore, some factors, like color, are naturally circular; a hue of zero is the same as a hue of one, creating a loop. Traditional methods often cut this loop open to fit it onto a flat line, which breaks the natural continuity of the data.

To address these issues, the researchers proposed a new framework that combines three elements: the topology, or the overall shape of the space; the identifications, which determine how different points connect or wrap around; and the local scales, which dictate how much visual change occurs at any given point. They realized that to truly separate these factors, the computer's internal map must mirror this complexity. They developed FactoMap, a system that learns a set of prototypes, or representative examples, arranged on a lattice that matches the underlying structure of the data. Instead of forcing the data into a rigid grid, FactoMap allows the lattice to take on the shape of a cone or a cylinder. In their experiments, they created a synthetic dataset called FactoShapes, featuring objects with varying hues, sizes, and positions. They found that when they used a standard rectangular grid to represent the data, the system failed to separate the factors cleanly. The map could not handle the circular nature of the color or the way the scale of the color changed with the object's size.

However, when the researchers supplied FactoMap with a lattice shaped like a cone, the results changed dramatically. In this structure, the circular direction of the lattice wrapped around to represent the hue, preserving its continuous nature without a break. Simultaneously, the cone's shape allowed the circumference of the circle to expand or contract as the lattice moved along the scale axis, perfectly matching the way the visual impact of color changed with size. This alignment allowed the system to learn a representation where the color and size factors were cleanly separated. The prototypes learned by the system organized themselves so that moving along one axis changed only the color, while moving along the other changed only the size, even though the underlying geometry was non-uniform. The study measured this success using standard metrics for disentanglement, showing that the matched cone structure significantly outperformed the mismatched grid.

The findings suggest that the path to better artificial intelligence lies not in forcing data into simple, uniform shapes, but in understanding and respecting the unique geometry of the world the data comes from. The researchers showed that statistical independence does not guarantee geometric simplicity. Factors can be sampled independently yet remain coupled through the mechanisms that generate the data. By building a map that can adapt its shape to these couplings—allowing for periodic loops, collapsing points, and varying scales—FactoMap achieves a level of clarity that rigid, flat models cannot. This approach does not just separate the factors; it does so by acknowledging that the space in which they live is not a flat sheet, but a dynamic, structured landscape. The work provides a concrete demonstration that to truly understand the factors of variation, one must first understand the shape of the space they inhabit.

Technical Summary: FactoMap for Disentangled Representations

Problem Statement
Disentangled representation learning aims to recover the underlying generative factors of a dataset such that manipulating a single factor corresponds to changing a single coordinate in the representation. While existing methods often rely on statistical independence or geometric assumptions like uniform local isometry (where equal changes in factor space produce equal changes in observation space, up to a fixed global scale), these assumptions frequently fail in practice.

The paper identifies two primary limitations in current geometric approaches:

Non-Uniform Geometry: Statistically independent factors can be geometrically coupled. For instance, in a rendering where hue and scale vary independently, the effect of changing hue on the observation (e.g., area) and the effect of changing scale (e.g., boundary) grow at different rates. This creates position-dependent anisotropy that cannot be corrected by a fixed rescaling of units.
Topological and Structural Insufficiency: Standard topology alone is insufficient to distinguish between spaces that are homeomorphic (topologically equivalent) but possess different factor organizations. For example, a square, a disk, and a cone are topologically similar, yet they support different factor geometries (e.g., a circular factor may retain constant extent, shrink, or collapse). Furthermore, local isometry cannot determine global properties like periodicity (closure) or the termination of factors.

Methodology: Factor-Space Structure and FactoMap
To address these issues, the authors introduce a new formalism called Factor-Space Structure and a corresponding learning algorithm, FactoMap (Factor-Space Topographic Map).

Factor-Space Structure ( $F_g$ ):
The authors define the factor-space structure as a triplet $F_g = (Z, \approx_g, c_g)$ , where:
- $Z = \prod Z_i$ is the product of factor domains (intervals $I$ or circles $S^1$ ).
- $\approx_g$ represents generator-induced identifications, defining which points in $Z$ map to the same observation (e.g., periodicity or collapse at a point).
- $c_g = (c_1, \dots, c_p)$ represents position-dependent scales, where $c_i(z) = \|\frac{\partial g}{\partial z_i}(z)\|_2$ measures the local effect of factor $i$ on the observation space.
  This formalism distinguishes spaces that share a topology but differ in how factors are identified or scaled.
FactoMap Architecture:
FactoMap is a structured prototype learner that realizes the factor-space structure as a topographic representation. It consists of three components:
- Factor-Space Lattice ( $\Lambda_F, d_F$ ): A finite lattice supplied by the user that encodes the specific topology (e.g., ring vs. line), identifications (e.g., collapsed fibers), and analytic distance functions reflecting the position-dependent scales.
- Prototype Field ( $W$ ): A set of learnable prototypes $w_k \in \mathbb{R}^m$ indexed by the lattice sites $k \in \Lambda_F$ .
- Topographic Learning Objective: The model learns by minimizing a self-organizing map (SOM) objective:
  $L_t(W) = \mathbb{E}_x \left[ \sum_{k \in \Lambda_F} H_{\sigma_t}(q_W(x), k) \|x - w_k\|^2 \right]$
  where $q_W(x)$ is the best-matching lattice site, and $H_{\sigma_t}$ is a neighborhood kernel based on the lattice distance $d_F$ . This encourages nearby lattice sites to represent nearby observations, effectively transferring the lattice's geometric and topological properties to the learned prototypes.

Key Contributions

Factor-Space Formalism: The paper proposes a rigorous separation of factor domains, generator-induced identifications, and factor-wise scales. This allows for the distinction of homeomorphic spaces with different geometric organizations.
Demonstration of Uniform Isometry Failure: The authors provide a theoretical proof (Proposition 2.1) and empirical evidence showing that statistically independent factors (specifically hue and scale) can exhibit position-dependent anisotropy. They demonstrate that no fixed diagonal re-weighting can make the metric uniformly Euclidean across a non-degenerate scale interval.
FactoMap Algorithm: A novel method that learns interpretable prototypes indexed by a lattice. By supplying a lattice that matches the underlying factor-space structure (including periodicity and non-uniform scaling), FactoMap achieves disentanglement where standard Euclidean or mismatched topological assumptions fail.
Controlled Evaluation: The introduction of FactoShapes, a synthetic dataset with controlled generative factors, allows for the isolation of how factor-space structure affects disentanglement.

Results
Experiments on FactoShapes demonstrate that matching the prototype domain to the underlying factor-space structure is critical for disentanglement:

Topology Matching: Using an open line lattice for a periodic factor (hue) introduces a "cut," separating adjacent hues. A ring lattice correctly restores adjacency, proving that global closure must be encoded in the lattice, not just learned locally.
Scale Matching: When hue and scale vary jointly, their image-space speeds follow different power laws ( $c_h \propto s$ and $c_s \propto \sqrt{s}$ ). A rectangular lattice fails to disentangle these factors, mixing them in the representation.
Conical Lattice Success: A "cone-structured" lattice, where the cyclic circumference varies with the scale axis (matching the analytical prediction), successfully disentangles hue and scale.
Quantitative Metrics: Using InfoMEC (a disentanglement metric), the matched cone lattice achieved high scores (InfoM $\approx$ 0.95, InfoE $\approx$ 0.82, InfoC $\approx$ 0.95), whereas a mismatched grid lattice performed poorly (InfoM $\approx$ 0.27, InfoE $\approx$ 0.54, InfoC $\approx$ 0.025).

Significance and Claims
The paper claims that statistical independence does not imply geometric separability. The underlying geometry of generative factors can be complex, involving non-uniform scales and specific topological identifications that standard Euclidean or fixed-topology models cannot capture.

FactoMap's significance lies in its ability to explicitly encode the factor-space structure (topology, identifications, and local scales) into the learning process. By aligning the lattice coordinates with the underlying factors, the method preserves factor continuity and enables disentanglement even when the data-generating process violates the assumption of uniform local isometry. The authors emphasize that while topology alone is insufficient, combining it with generator-induced identifications and position-dependent scales provides a necessary framework for representing complex generative factors.

Technical Summary: FactoMap for Disentangled Representations

More like this