🤖 machine learning

Do Neural Networks Learn Structure-Preserving Maps? A Case Study in Latent-to-Hilbert Embeddings

This paper demonstrates that neural networks can learn structure-preserving maps from compressed latent spaces to Hilbert-space representations that are effectively linear and rank-4, yet suggests that classical kernel methods may outperform tuned MLPs for such approximately linear tasks.

Original authors: Muhammad Adnan Shahzad

Published 2026-10-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammad Adnan Shahzad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of modern computing, machines are increasingly taught not just to recognize patterns, but to understand the hidden geometry of data. Imagine a vast library where every book is represented not by its cover, but by a unique set of coordinates in a multi-dimensional space. In this space, the distance between two points reflects how similar the books are, and the angle between them reveals how their contents relate. This is the essence of "representation learning," a field where artificial intelligence systems compress complex information into simpler, manageable forms. A central question for researchers is whether these compressed forms can be translated back into a different, highly structured mathematical language without losing the delicate relationships between the original items. Specifically, scientists are interested in "structure-preserving maps," which are transformations that keep these geometric relationships intact, ensuring that if two items were close together in the original data, they remain close together in the new representation. This capability is crucial for fields ranging from aligning different types of data to bridging the gap between classical computers and the emerging field of quantum computing, where information is stored in the states of subatomic particles.

A researcher set out to test whether a neural network—a type of computer program modeled after the human brain—could learn to perform such a translation. They designed an experiment using a well-known dataset of handwritten digits, compressing images of numbers into a tiny, eight-dimensional summary. From this summary, they created a target representation based on the mathematical rules governing a small system of quantum particles. The goal was to see if the neural network could learn to map the compressed summary directly to this quantum-style representation, preserving the inner relationships between the numbers. The researcher compared the neural network's performance against simpler, classical mathematical tools to see which approach was better at maintaining the structural integrity of the data.

The results revealed a surprising simplicity in how the neural network solved the problem. Despite the complex architecture of the network and the nonlinear nature of the target representation, the mapping it learned was almost entirely linear. When the researcher analyzed the network's behavior, they found that a simple straight-line equation could describe its actions with an accuracy of 91 percent, nearly matching the 98 percent accuracy achieved when mapping directly to the ideal target. This suggests that the network did not need to perform complex, twisting calculations to solve the task; instead, it found a direct, efficient path. Furthermore, the direction in which the network pushed the data was not determined by the features of the input images themselves, but by the geometry of the target quantum system. The network effectively ignored the most obvious patterns in the input data and instead focused entirely on the specific directions required to match the target structure.

Perhaps the most counterintuitive finding concerned the importance of the different components within the network's solution. The mathematical analysis of the network's output showed that two of its internal directions were much stronger than the other two. Intuitively, one might expect the weaker directions to be negligible and safe to discard. However, the researcher found that removing even the weakest direction caused the quality of the mapping to degrade significantly, increasing the error by a factor of nearly six. This demonstrated that every single direction the network learned was essential for preserving the geometric structure, regardless of how small its contribution appeared to be. The network had identified a precise, four-dimensional subspace that was distinct from the standard mathematical basis used to create the target, and it remained stable across different training runs, consistently finding the same primary direction while varying slightly in the others.

When the researcher compared the neural network to a classical method known as kernel ridge regression, the simpler tool proved superior. While the neural network required extensive tuning of its internal settings and still produced a certain level of error, the classical method achieved a lower error rate with less effort. This suggests that for tasks where the underlying relationship is approximately linear, complex neural networks may not be the most efficient choice. The study indicates that in these specific scenarios, classical mathematical approaches can serve as a stronger baseline, outperforming the more elaborate neural architectures. The findings do not imply that neural networks are useless for all structure-preserving tasks, particularly when the relationships are highly nonlinear, but they do highlight that for this specific type of problem, a simpler tool was the more effective solution.

Ultimately, the study provides a clear picture of how a neural network learns to translate between compressed data and structured mathematical spaces. It learned a mapping that was linear, driven by the target rather than the input, and reliant on every single dimension it discovered. By demonstrating that a classical method could outperform a tuned neural network in this setting, the research offers a practical guideline for future work: when the goal is to preserve geometric structure in a way that is roughly linear, researchers should consider simpler, classical methods before turning to more complex neural approaches. This insight helps clarify the boundaries of when deep learning is necessary and when more straightforward mathematical tools are sufficient to capture the essential structure of the data.

Technical Summary: Do Neural Networks Learn Structure-Preserving Maps?

Problem Statement

This paper investigates whether a neural network can learn a mapping $f(z) = \psi$ from a compressed latent space $z$ to a target Hilbert-space representation $\psi$ that preserves the geometric structure (specifically, inner products) between points. While existing literature often assumes a fixed target representation and measures proximity (e.g., via Canonical Correlation Analysis or Procrustes alignment), this study asks a distinct question: can a network learn a mapping into a specified target space while actively preserving the underlying geometric structure? The problem is situated at the intersection of representation similarity analysis, kernel methods, and quantum machine learning, specifically addressing classical-to-quantum mappings where classical data is encoded into quantum product states via angle encoding.

Methodology

The author employs a controlled experimental setup using the MNIST dataset and a simulated $n$ -qubit system:

Latent Space Construction: An 8-dimensional autoencoder is trained on 5,000 MNIST images to compress $28 \times 28$ images into latent vectors $z \in \mathbb{R}^8$ . This encoder is fixed and not fine-tuned during subsequent mapping experiments.
Target Hilbert Space: The target is an $n$ -qubit product state $\psi(\theta)$ , constructed via the tensor product of single-qubit rotations $R_y(\theta_i)$ . The state is defined by real amplitudes derived from a vector of $n$ angles $\theta$ .
Target Generation: The target angles $\theta(z)$ are deterministically generated from the latent vector $z$ by projecting $z$ onto the top- $n$ principal components of the training latents, applying a sigmoid transformation, and rescaling to $[0, \pi]$ .
Models Evaluated: Four models are trained to map $z$ $z$ to $\theta$ $θ$ (from which $\psi$ $ψ$ is reconstructed):
- Linear Regression: A simple linear map $Wz + b$ .
- MLP (MSE only): A two-layer Multi-Layer Perceptron trained on Mean Squared Error.
- MLP (MSE + IP Loss): An MLP trained on a combined loss including an inner-product preservation term.
- Kernel Ridge Regression (KRR): A classical method using an RBF kernel to regress directly to the true target angles.
Metrics: Performance is evaluated using Inner-Product Preservation Error (IP err), Reconstruction Error (MSE), and Pairwise Distance Preservation Error.

Key Findings and Results

1. Approximate Linearity of the Learned Map

Despite the target angles being generated via a nonlinear sigmoid transformation, the learned mapping is well-approximated by a linear function over the observed latent distribution.

Linear regression from $z$ to the true target angles achieves $R^2 = 0.98$ .
Linear regression from $z$ to the MLP's recovered angles achieves $R^2 = 0.91$ .
The small gap suggests the MLP effectively learns a linear map, with residual nonlinearity consistent with the approximately linear range of the sigmoid function.

2. Necessity of All Singular Directions

The learned map is genuinely rank-4 (for $n=4$ ), despite a singular value spectrum showing two dominant values and two small values (approx. 5.5, 5.3, 1.3, 0.8).

Truncating the map to rank 3 degrades IP error by a factor of 2.5.
Truncating to rank 2 degrades IP error by a factor of 4.2.
Truncating to rank 1 degrades IP error by a factor of 5.9.
Conclusion: Every singular direction carries essential geometric information; magnitude-based rank truncation is unsafe for this class of problems.

3. Target-Driven Primary Direction

The geometry of the learned map is driven by the target, not the input.

The primary direction of the learned map (first right singular vector) has a cosine similarity of 0.002 with the input's principal direction (essentially orthogonal).
It has a cosine similarity of 0.989 with the target's principal direction lifted back into the input space.
The learned subspace does not coincide with the PCA basis used to construct the target. While the primary direction is stable across random seeds (varying by $<6^\circ$ ), higher-rank directions diverge significantly between seeds.

4. Superiority of Kernel Methods

In this specific setting, classical kernel methods outperform tuned neural networks.

KRR Performance: Best configuration ( $\alpha=10^{-4}, \gamma=0.1$ ) achieved IP err = 0.0144 and MSE = 0.0013.
MLP Performance: Best configuration (512 hidden units, 2000 epochs, $w_{IP}=1$ ) achieved IP err = 0.0197 and MSE = 0.0015.
KRR outperforms the best MLP by a factor of 1.37 in IP error. The Pareto frontier of the MLP lies strictly above the KRR result, indicating KRR achieves both lower reconstruction and lower inner-product error.

Significance and Claims

The paper claims that for approximately linear, structure-preserving mappings from compressed latent spaces to Hilbert-space representations:

Neural networks are not strictly necessary: Classical kernel methods (specifically KRR with an RBF kernel) serve as a stronger, simpler baseline than tuned MLPs.
Geometry is target-driven: The network learns to identify input directions that are relevant to the target geometry rather than simply projecting the input's principal components.
Rank sensitivity: Structure preservation relies on the full rank of the map, even when singular values suggest lower-rank approximation is possible.

The author concludes that while neural networks are capable of learning these mappings, their advantage may be limited to settings with genuinely nonlinear targets (e.g., entangled quantum states). For the approximately linear cases studied here, simpler methods suffice and should be considered first. The findings are presented as a specific instance of the broader phenomenon where kernel methods perform competitively or better than finite-width networks on small datasets, contributing to the discussion of when neural networks are the appropriate tool for representation learning.