🤖 machine learning

Understanding DNNs in Feature Interaction Models: A Dimensional Collapse Perspective

This paper proposes a novel "dimensional collapse" perspective to resolve the debate on DNNs in feature interaction models, demonstrating through experiments and gradient-based theory that DNNs effectively mitigate embedding dimensional collapse to enhance representation robustness.

Original authors: Jiancheng Wang, Mingjia Yin, Hao Wang, Enhong Chen

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiancheng Wang, Mingjia Yin, Hao Wang, Enhong Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Why Do We Need "Deep" Neural Networks?

Imagine you are trying to predict if a user will click on an ad. You have a lot of information about them: their age, the time of day, the device they are using, and what they like. In the world of recommendation systems, these are called features.

For a long time, researchers have used two main tools to figure out how these features work together:

Explicit Interaction Models (The "Mathematical Calculator"): These are like strict calculators that look at specific pairs of features (e.g., "Age" + "Time of Day") and multiply them to find a pattern. They are good, but they can get stuck in a rut.
Deep Neural Networks (DNNs) (The "Smart Brain"): These are complex, multi-layered systems that try to learn hidden, complicated patterns on their own.

The Debate:
There has been a huge argument in the research community. Some people say, "DNNs are amazing because they can find super-complex, hidden patterns that calculators miss!" Others say, "Actually, DNNs are terrible at even doing simple math (like multiplying two numbers), so they probably can't find those complex patterns either."

The New Discovery:
This paper doesn't join that argument. Instead, it looks at the problem from a completely different angle: Dimensional Collapse.

The Core Concept: "Dimensional Collapse"

Imagine you have a giant, colorful 3D sculpture made of clay. This sculpture represents all the information your computer has about a user. It has height, width, and depth (dimensions).

Dimensional Collapse is what happens when that 3D sculpture accidentally gets squashed flat into a 2D piece of paper.

Before the crash: The sculpture is rich, detailed, and can be viewed from many angles.
After the crash: It's flat. You've lost a lot of information. The computer is trying to understand a complex 3D world using only a flat 2D map. It's like trying to navigate a city using a drawing on a napkin instead of a GPS.

The paper argues that "pure" interaction models (the calculators) tend to squash this 3D sculpture flat. They force all the information into a tiny, narrow space, making the model less robust and less accurate.

The Solution: The DNN as a "Squash-Preventer"

The authors discovered that adding a Deep Neural Network (DNN) to these models acts like a structural support beam or a spring that keeps the sculpture from collapsing.

They tested two ways of adding this "spring":

Parallel DNN: Like having a second, independent worker standing next to the calculator, looking at the same data and shouting out extra insights.
Stacked DNN: Like putting a smart filter on top of the calculator's output to smooth things out.

The Result:
Whether they used the "Parallel" or "Stacked" method, the DNN successfully stopped the sculpture from flattening. The data stayed "3D" (or high-dimensional), meaning the model could see more details and make better predictions.

Breaking Down the DNN: Two Parts, One Job

The paper also took the DNN apart to see which part was doing the heavy lifting. A DNN has two main ingredients:

Linear Parts: These are like straight lines or simple stretching.
Non-Linear Parts (Activations): These are like bends, curves, and twists.

The Finding:
Both parts are necessary to keep the sculpture from collapsing.

The Linear parts act like a wide net, catching more of the data and spreading it out so it doesn't bunch up.
The Non-Linear parts act like a sculptor, twisting and turning the data to ensure it fills the space evenly, rather than getting stuck in a corner.

The "Why": The Gradient Analogy

To understand how this works, the authors looked at the "learning process" (called gradients).

Without DNN: Imagine the calculator is trying to learn by walking. But its legs are tied together. It can only move in a few specific directions (forward, backward, left, right). It can't explore the whole field. This limited movement causes the "collapse."
With DNN: The DNN unties the legs and gives the walker a map of the whole field. It allows the model to take steps in directions it couldn't reach before. This freedom of movement keeps the data spread out and healthy.

Summary of Claims

The Problem: Feature interaction models tend to squish their data into a tiny, low-dimensional space (Dimensional Collapse), which hurts performance.
The Fix: Adding a Deep Neural Network (either parallel or stacked) prevents this squishing.
The Mechanism: The DNN changes the way the model learns (the gradients), allowing it to explore a wider variety of directions rather than getting stuck in a narrow path.
The Components: Both the straight-line math (linear) and the curved math (non-linear) inside the DNN are needed to keep the data robust.

What the paper does NOT claim:
The paper does not claim that DNNs are better because they find "hidden high-order interactions" (the old debate). Instead, it claims they are better because they keep the data from collapsing. It also does not discuss future applications or clinical uses; it strictly analyzes how these models work on ad-click prediction datasets (Avazu and Criteo).

1. Problem Definition

In Click-Through Rate (CTR) prediction, Deep Neural Networks (DNNs) are widely integrated with explicit feature interaction models (e.g., FM, CrossNet) to capture complex relationships. However, the specific role of DNNs remains debated:

Conventional View: DNNs are believed to implicitly capture high-order feature interactions that explicit models miss.
Contradictory Evidence: Recent studies suggest DNNs (specifically MLPs) struggle to learn simple dot products (second-order interactions) and may underperform compared to explicit high-order models.
The Gap: There is insufficient theoretical understanding of why DNNs improve performance in these hybrid architectures despite their limitations in learning specific interaction types.

The authors propose that the primary benefit of DNNs is not just capturing interactions, but mitigating "Dimensional Collapse" in feature embeddings. Dimensional collapse occurs when embeddings degenerate into a narrow, low-dimensional subspace, characterized by a rapid decay in the singular value spectrum of the representation matrix.

2. Methodology

The authors employ a combination of empirical evaluation, ablation studies, and gradient-based theoretical analysis.

A. Evaluation Protocols

Datasets: Avazu and Criteo (standard CTR benchmarks).
Baselines:
- Pure Interaction Models: Factorization Machine (FM) and CrossNet.
- Hybrid Models: DeepFM/NFM (FM + DNN) and DCNv2 (CrossNet + DNN).
Metrics:
- Predictive Performance: AUC (Area Under the Curve).
- Dimensional Robustness: RankMe (a metric based on the Shannon entropy of normalized singular values) and Singular Value Spectrum analysis. A higher RankMe and a flatter spectrum indicate less collapse and more robust representations.

B. Experimental Design

The study investigates two DNN integration strategies:

Parallel DNN (p-DNN): The DNN receives the same raw input as the interaction module (e.g., DeepFM).
Stacked DNN (s-DNN): The DNN processes the output of the interaction module (e.g., NFM).

C. Ablation Study

To isolate the contributions of DNN components, the authors progressively augment the base models:

Linear Component: Adding only linear projections (Linear-MLP) without non-linear activations.
Non-Linear Component: Adding non-linear activation functions (e.g., ReLU) on top of the linear projections.

D. Theoretical Analysis

The authors derive the gradients of the embeddings with respect to the loss function for both pure interaction models and DNN-enhanced models. They analyze how the gradient flow influences the update direction of the embedding vectors.

3. Key Contributions

Novel Perspective: This is the first work to analyze the effectiveness of DNNs in feature interaction models through the lens of dimensional collapse.
Empirical Discovery: Demonstrated that both parallel and stacked DNNs significantly alleviate dimensional collapse in pure interaction models (FM and CrossNet), evidenced by flatter singular value spectra and higher RankMe scores.
Component Analysis: Identified that both the linear projection layers and the non-linear activation functions contribute to mitigating collapse. Linear projections expand effective dimensionality, while non-linearities further flatten the spectrum.
Gradient Mechanism: Provided a theoretical explanation showing that DNNs introduce orthogonal gradient components and dimension-aware weighting, allowing embeddings to escape low-rank subspaces and update in a broader vector space.

4. Key Results

A. Predictive Performance & Dimensional Robustness

AUC: Adding DNNs consistently improved AUC across both datasets.
RankMe (Robustness):
- On the Avazu dataset, adding p-DNN to FM increased RankMe from 89.12 to 166.78.
- On the Criteo dataset, p-DNN increased FM's RankMe by 158%, effectively rescuing the model from a low-rank state.
Spectral Analysis: Pure models (FM/CrossNet) showed sharp spectral decay (indicating collapse). Integrating DNNs flattened the spectrum, maintaining higher singular values across the top 260 dimensions.

B. Component Ablation

Linear DNN: Introducing linear projections alone significantly lifted the singular spectrum, expanding the effective dimensionality.
Non-Linear Activations: Adding non-linearities further flattened the spectrum, particularly in the intermediate-to-large singular components (indices 25–200), enhancing representational capacity.

C. Gradient Analysis Findings

Pure FM: The gradient update for an embedding $v_i$ is strictly restricted to the subspace spanned by its interacting features $\{v_j\}$ . This constraint forces embeddings into a low-rank subspace, causing collapse.
DNN-Enhanced Models:
- Parallel DNN: Introduces a highly non-linear term ( $g_{DNN}$ ) that provides orthogonal gradient components. This allows embeddings to update in directions outside the low-rank interaction subspace.
- Stacked DNN: Acts as a dimension-aware weighting vector, decoupling dependencies and preventing dimensions from converging toward a single dominant direction.
Empirical Gradient Study: Tracking gradients during training showed that FM gradients remain concentrated in the top-30 singular components, whereas DNN-enhanced models exhibit smoother, more diverse gradient distributions with higher RankMe scores throughout training.

5. Significance

This paper fundamentally shifts the understanding of DNNs in recommendation systems. Instead of viewing DNNs merely as "black boxes" for capturing implicit high-order interactions, the authors prove their critical role as regularizers against dimensional collapse.

Theoretical Insight: It resolves the debate on DNN limitations by showing that their value lies in preserving the dimensional robustness of embeddings, which is essential for learning complex patterns in sparse data.
Practical Guidance: The findings suggest that when designing feature interaction models, ensuring the preservation of embedding dimensionality (via linear and non-linear DNN components) is as important as the explicit interaction mechanism itself.
Future Directions: The gradient-based analysis provides a new tool for diagnosing representation quality and designing architectures that explicitly prevent collapse in high-cardinality categorical data.