Technical Summary: Learning Coherent Representations: A Topological Approach to Interpretability

Problem Statement

Deep neural networks (DNNs) often learn representations where individual latent features lack interpretable meaning. While classification training successfully separates class manifolds in latent space, it does not guarantee that individual neurons correspond to human-understandable concepts. In unsupervised settings like auto-encoders, the issue is exacerbated: without class labels to guide separation, networks may learn representations where semantically related samples are scattered, and individual features activate on disconnected, incoherent subsets of the data manifold.

Existing regularization techniques, such as sparsity (L1), encourage features to activate on few samples but fail to ensure those samples are geometrically related. A sparse feature may still fire on scattered regions of the data manifold. Conversely, biological neural circuits (e.g., grid cells and head direction cells) achieve interpretability through locality: neurons respond to contiguous regions of the state space. This paper posits that locality is a design principle for interpretable neural codes and seeks to formalize this principle to impose it on artificial neural networks.

Methodology

1. Theoretical Framework: Coherent Matrices

The authors introduce coherence, a geometric property inspired by neural coding and grounded in topological data analysis (TDA). They model the latent space of a network with non-negative activations as a non-negative matrix $M$, where rows represent samples and columns represent features.

A matrix is defined as $\epsilon$-coherent if it satisfies two conditions:

1. $\epsilon$-locality: Each row (sample) attends to a geometrically clustered set of columns (features), and vice versa. This is quantified using Fréchet variance. Specifically, the barycentric map $\phi$ (mapping a sample to the weighted average of features) and $\psi$ (mapping a feature to the weighted average of samples) must have low variance relative to the actual features/samples.
2. $\epsilon$-covering: Every sample is well-described by some feature, and every feature is needed by some sample. This ensures no part of the space is left uncovered or redundant.

Topological Guarantee:
The paper proves that if a matrix is $\epsilon$-coherent and the barycentric maps are Lipschitz continuous, the Vietoris-Rips filtrations of the samples and features are $\epsilon^{1/2}$-interleaved. This implies that the sample space and the feature space share compatible topological structures. For instance, if data lies on a circle, coherent features must tile that circle into contiguous arcs, ensuring the feature space mirrors the sample space's topology.

2. The COH Objective Function

To enforce coherence during training, the authors derive COH, a differentiable loss function based on Fréchet variance.

• Normalization: The method uses squared L1 normalization to treat rows and columns as probability distributions.
• Loss Components: The loss penalizes the variance (locality) and covering quantities for both rows and columns.
• Implementation: The objective is added to any architecture with non-negative activations (e.g., using Softplus). It computes barycentric maps and their variances, aggregating the top-$k$ worst offenders to ensure robustness against outliers.

Key Contributions

1. Definition of Coherence: The authors formally define $\epsilon$-coherence for non-negative matrices, linking it to the interleaving of Vietoris-Rips filtrations. This provides a theoretical guarantee that the latent sample space and latent feature space share compatible topology.
2. Differentiable Objective (COH): They introduce COH, a practical loss function that enforces locality and covering constraints. Unlike sparsity, which bounds the number of activations, COH bounds the geometric connectivity of activations.
3. Empirical Validation: The method is validated in three distinct settings:
   • Synthetic Data: Recovering the topology of disjoint circles.
   • Rotated MNIST: Learning interpretable features for rotated digits.
   • Language Models: Applying coherence to BERT token embeddings to yield interpretable features aligned with human-readable categories.

Experimental Results

Synthetic and Rotated MNIST Auto-Encoders

• Topology Recovery: In the "Two Circles" toy experiment, standard auto-encoders and L1-regularized models failed to produce a feature space with the correct circular topology. Only the COH model recovered the expected disjoint circular topology in both the sample and feature spaces.
• Single Digit (Rotated MNIST): For a single rotated digit (6), COH achieved 100% tuned features (features activating on coherent arcs of rotation), compared to 63% for L1. COH features were less sparse but significantly more interpretable regarding the rotation angle.
• Double Digit (Rotated MNIST): In a more complex setting with two digits, COH consistently produced coherent latent spaces. While interpretability metrics (like purity) varied across seeds due to the existence of multiple valid coherent solutions (e.g., separated circles vs. merged circles), the coherence metrics (Locality and Covering) remained stable. Combining COH with L1 sparsity successfully guided the model toward a disentangled solution with reliable class separation.

BERT Token Embeddings

• Setup: A small BERT model was trained with non-negative token embeddings (via Softplus) and the COH objective.
• Interpretability:
  • Overlap: Features in the COH model showed significantly higher overlap with the nearest-neighbor token neighborhoods of a standard (signed) BERT model (Mean Overlap: 0.45 vs. 0.22 for Softplus-only).
  • Human-Readable Categories: Using an LLM (Claude Opus) to judge interpretability, 87.6 out of 256 features in the COH model were deemed interpretable, compared to 0 for the Softplus-only baseline.
  • Categories Identified: COH features aligned with specific semantic categories including years, kinship terms, place names, units of measurement, hedging adverbs, directional prepositions, and specific verb types (e.g., motion, cognition, past tense).

Significance and Claims

The paper claims to provide a rigorous framework for coupling latent samples and latent features so that they share compatible topology. The significance lies in shifting the focus from individual feature interpretability to the interpretability of the feature space as a whole.

• Geometric Guarantee: The work demonstrates that enforcing geometric connectivity (coherence) yields features that are not just sparse but structurally aligned with the data manifold.
• Complementarity: The authors note that coherence is complementary to other objectives; for example, sparsity can be used to select among multiple valid coherent solutions without sacrificing the geometric guarantee.
• Limitations: The authors modestly acknowledge that the Lipschitz assumption on barycentric maps is verified empirically rather than enforced. They also note that the method does not control which specific coherent solution is found in complex multi-class settings (e.g., whether digits merge or separate), though it guarantees the solution is coherent.

The work connects topological data analysis, neuroscience, and representation learning, offering both a theoretical foundation (via Dowker duality analogues and interleaving) and a practical tool for learning interpretable latent spaces without requiring explicit supervision.