Technical Summary: Symmetric Divergence and Normalized Similarity: A Unified Topological Framework for Representation Analysis

Problem Statement

Topological Data Analysis (TDA) offers an intrinsic perspective for comparing neural network representations, yet existing macroscopic methods face significant limitations. Specifically, the Representation Topology Divergence (RTD) framework and its variants suffer from two critical issues:

1. Theoretical Incompleteness and Asymmetry: The commonly used "symmetric" RTD is defined as a brute-force average of two directional values ($RTD(w, \tilde{w})$ and $RTD(\tilde{w}, w)$). These values can differ dramatically without a clear theoretical explanation for the asymmetry. Furthermore, the dual variant, Max-RTD, proposed to enrich gradient information, lacks a fully characterized theoretical relationship with RTD.
2. Lack of Normalization: RTD and RTD-lite produce unbounded positive scores that depend heavily on sample size and intrinsic distance scales. This makes cross-scenario benchmarking (e.g., comparing different models, layers, or datasets) unreliable and difficult to interpret. Existing attempts to rescale distances (e.g., using the 0.9-quantile) are heuristic and insufficient for heterogeneous settings.

These limitations hinder the ability to perform both fine-grained structural diagnosis and robust, standardized evaluation of neural representations.

Methodology

The authors propose a unified topological toolkit comprising two complementary components: Symmetric Representation Topology Divergence (SRTD) for structural diagnosis and Normalized Topological Similarity (NTS) for standardized evaluation.

1. Symmetric Representation Topology Divergence (SRTD)

To address the asymmetry and theoretical gaps in the RTD family, the authors introduce SRTD.

• Concept: Instead of averaging two directional divergences, SRTD directly measures the topological discrepancy between the union and intersection of the Vietoris-Rips filtrations of two point clouds.
• Construction: It utilizes a symmetric auxiliary matrix ($M_{sym}$) constructed from the element-wise minimum ($min(w, \tilde{w})$) and maximum ($max(w, \tilde{w})$) of the distance matrices. The divergence is defined as the sum of barcode lengths derived from the persistent homology of this auxiliary graph.
• Theoretical Unification: SRTD provides a unified framework where the asymmetric measures (RTD and Max-RTD) are decomposed into a large, shared symmetric component (SRTD) and smaller, "private" components unique to individual filtrations.
• Efficiency (SRTD-lite): Following the RTD-lite approach, the authors define SRTD-lite, which approximates the full topological divergence using Minimum Spanning Tree (MST) weights. It is defined as the difference between the MST weights of $max(w, \tilde{w})$ and $min(w, \tilde{w})$.
• Key Property: The authors prove that for the lite version, $Max\text{-}RTD\text{-}lite + RTD\text{-}lite = SRTD\text{-}lite$, establishing a strict hierarchical ordering where SRTD-lite serves as a balanced, median measure.

2. Normalized Topological Similarity (NTS)

To enable robust cross-scenario benchmarking, the authors propose NTS, a metric bounded between -1 and 1.

• Motivation: Unlike divergence measures that sum barcode lengths (which are sensitive to outliers and scale), NTS focuses on the rank correlation of hierarchical merge orders.
• Mechanism:
  • It identifies the set of "core pairs" defined by the union of edges in the MSTs of both representations.
  • It computes the Spearman's rank correlation between the merge times (or edge distances) of these core pairs in the two representations.
  • Two variants are defined: NTS-E (based on edge distances) and NTS-M (based on merge times).
• Properties: NTS is scale-invariant and robust to sample size variations. It captures the topological structure by comparing the order in which connected components merge, rather than the absolute magnitude of distances.

Key Contributions

1. Completion of the RTD Framework: The introduction of SRTD resolves the theoretical asymmetry of prior RTD variants. It consolidates diagnostic information into a single, symmetric cross-barcode signature, offering a more efficient optimization objective without the overhead of dual directional computations.
2. Introduction of NTS: The proposal of NTS overcomes the unbounded nature of topological divergences. By leveraging rank-based normalization, it provides a standardized similarity score that is comparable across different datasets, model architectures, and experimental pipelines.
3. Theoretical Characterization: The paper establishes rigorous mathematical relationships between RTD, Max-RTD, and SRTD using mapping cones and long exact sequences in homology, clarifying the role of Max-RTD and the source of asymmetry.
4. Efficient Algorithms: The authors provide lightweight variants (SRTD-lite and NTS-E) that operate in $O(n^2(\alpha_{uf}(n) + d))$ time, making them scalable for large-scale representation analysis.

Experimental Results

The toolkit was evaluated across synthetic and real-world deep learning settings:

• Synthetic Hierarchical Shifts: In experiments involving Gaussian clusters and UMAP embeddings, NTS and SRTD families correctly captured increasing structural dissimilarity. In contrast, CKA was largely insensitive to these changes, and RTD-lite produced anomalous, inverted trends, highlighting the necessity of the $max(w, \tilde{w})$ component and normalization.
• CNN Layer-wise Analysis: On a TinyCNN trained on CIFAR-10, NTS and CKA exhibited consistent, interpretable near-diagonal similarity landscapes (similarity decaying with depth). However, the unnormalized RTD family failed to recover this monotonic trend, producing irregular heatmaps. Crucially, topological measures (NTS and SRTD-lite) detected a sharp structural transition at the final pooling layer—a functional shift missed by geometric baselines like CKA.
• Large Language Model (LLM) Analysis: Applied to LLM families (Qwen, InternLM, Baichuan, Llama) on TruthfulQA and ToxiGen datasets:
  • Intra-Model: NTS revealed consistent hierarchical "fingerprints" within model families, whereas CKA often produced saturated or inconsistent heatmaps due to distance saturation.
  • Inter-Model: NTS provided a sharper, more discriminative view of model relationships compared to CKA, which suffered from score saturation (clustering near 1.0).
  • Distillation Case: In a case study of DeepSeek-R1-Ds (distilled from Qwen2.5-Math), CKA yielded a low similarity score failing to reflect the lineage, while NTS successfully identified high structural similarity, suggesting topological merge orders are more robust to distillation-induced geometric shifts.

Significance and Claims

The paper claims to offer a rigorous, topology-aware perspective that complements existing geometric measures like CKA.

• For Diagnosis: SRTD and SRTD-lite provide fine-grained, interpretable evidence of structural discrepancies and serve as effective loss terms for optimization (e.g., in autoencoders), achieving state-of-the-art performance within the RTD class with improved efficiency.
• For Benchmarking: NTS offers a reliable, scale-invariant metric for comparing representations across heterogeneous settings, overcoming the interpretability issues of unnormalized divergences.
• Limitations: The authors note that NTS is currently an analysis-only measure and is non-differentiable, preventing its direct use in model optimization. Future work is suggested to develop a differentiable formulation. Additionally, the lightweight variants rely primarily on 0-dimensional features, trading off higher-order topological insights for scalability.

The work positions itself as a foundational step toward understanding neural representations through a unified topological lens, bridging the gap between fine-grained structural diagnosis and robust, standardized evaluation.