🤖 machine learning

Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation

To address the embedding collapse and limited expressivity in the RankMixer architecture, this paper proposes RankElastor, a novel recommendation model featuring parameterized full mixing and GLU-improved P-FFNs that stabilize representation spectra and enable robust dense scaling.

Original authors: Guoming Li, Shangyu Zhang, Junwei Pan, Wentao Ning, Jin Chen, Gengsheng Xue, Chao Zhou, Shudong Huang, Haijie Gu, Menglin Yang

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guoming Li, Shangyu Zhang, Junwei Pan, Wentao Ning, Jin Chen, Gengsheng Xue, Chao Zhou, Shudong Huang, Haijie Gu, Menglin Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The Recommendation Engine's Dilemma

Imagine you are running a massive library (a recommendation system) that suggests books to millions of readers. To do this, the library assigns a unique "ID card" (an embedding) to every book and every reader. These ID cards contain a lot of information.

Recently, a new, very smart librarian named RankMixer was hired. RankMixer is great at organizing these ID cards. It takes the cards, shuffles them around to find connections, and then runs them through a processing machine to make better suggestions.

However, the researchers in this paper discovered a hidden problem with RankMixer. As the library grows larger and the ID cards get more complex, the cards start to lose their individuality. They all begin to look the same, like a stack of identical photocopies. In technical terms, the system suffers from "Embedding Collapse." The information gets squashed into a tiny, flat space, and the library stops being able to tell the difference between a mystery novel and a cookbook, even though they are very different.

The Problem: The "Sawtooth" Rollercoaster

The researchers looked closely at how RankMixer processes these ID cards layer by layer. They found a strange pattern, like a rollercoaster that goes up and down but slowly drifts lower over time:

The Shuffling (Token Mixing): RankMixer first shuffles the cards. This is like spreading out a deck of cards on a table. This step actually helps! It spreads the information out and makes the ID cards look more diverse (the "Expand" part).
The Processing (P-FFN): Next, the cards go through a processing machine. Unfortunately, this machine tends to crush the cards back down into a flat pile. It shrinks the diversity (the "Shrink" part).

In the old RankMixer, the "Shrink" step was too strong. Even though the shuffling tried to expand the information, the processing machine crushed it back down. The result was a "damped oscillation"—a wobbly line that slowly trends toward a flat, useless state. The library was expanding its potential but immediately shrinking it back down.

The Solution: RankElastor

To fix this, the authors built a new librarian called RankElastor. Their motto is "Expand More, Shrink Less." They made two specific upgrades to the library's workflow:

1. The "Master Shuffler" (Parameterized Full Mixing)

The Old Way: RankMixer used a rigid rule to shuffle cards. It was like a machine that could only swap blocks of 10 cards at a time. It was efficient, but it couldn't make fine, detailed adjustments.
The New Way: RankElastor uses a "Master Shuffler." This is a learnable, flexible system that can mix every single card with every other card in a detailed way.
The Analogy: Imagine trying to mix a salad. The old way was like using a giant spoon that could only scoop up big chunks of lettuce and tomatoes. The new way is like using a pair of chopsticks that can pick up individual grains of rice and mix them perfectly. This allows the system to create much richer, more diverse ID cards that don't collapse easily.

2. The "Smart Processor" (GLU-improved P-FFNs)

The Old Way: The processing machine used a standard activation function (GELU). Think of this as a light switch that is either ON or OFF, or a dimmer that sometimes gets stuck. It tended to crush the information too hard.
The New Way: RankElastor swapped this for a GLU (Gated Linear Unit) processor.
The Analogy: Imagine the old processor was a heavy door that slammed shut, blocking most of the light. The new GLU processor is like a smart window with a dimmer switch and a gate. It can let the right amount of light through and control the flow more precisely. It acts as a "gatekeeper" that prevents the information from being crushed flat, keeping the ID cards distinct and useful.

The Results: A Healthier Library

The researchers tested RankElastor on two massive, real-world datasets (Criteo and Avazu), which are like huge catalogs of online ads and user clicks.

Better Recommendations: RankElastor made better predictions than the old RankMixer and other top competitors. It improved the accuracy of recommendations (measured by AUC) by a small but statistically significant amount. In the world of massive recommendation systems, even a tiny improvement is a huge win.
No More Collapse: When they looked at the "Effective Rank" (a measure of how diverse the ID cards are), RankElastor kept the cards much more diverse. Instead of the rollercoaster drifting down to a flat line, the cards stayed "bouncy" and varied throughout the entire process.
Scaling Up: When they made the library bigger (adding more layers or wider processing), RankElastor got better and better. The old RankMixer struggled to scale up without collapsing, but RankElastor handled the growth gracefully.

Summary

The paper argues that to build better recommendation systems, we need to stop the information from getting squashed. By replacing rigid shuffling with flexible mixing and using smarter processing gates, RankElastor ensures that the system "expands" its understanding of data more than it "shrinks" it. This keeps the recommendations fresh, diverse, and accurate, even as the system grows to massive sizes.

Technical Summary: Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation

1. Problem Statement

The paper addresses the challenge of embedding collapse in the context of scaling dense recommendation models. While recent architectures like RankMixer have demonstrated success in scaling recommendation systems by alternating between token mixing and per-token feedforward networks (P-FFNs), they suffer from a fundamental limitation: the learned representations tend to concentrate in low-rank subspaces.

This phenomenon, known as embedding collapse, limits the expressivity of the model and underutilizes the expanded representation space provided by scaling. The authors identify that RankMixer's specific architecture induces a "damped oscillatory trajectory" in effective rank across layers:

Token mixing modules slightly increase the effective rank.
P-FFN modules subsequently contract the effective rank.

While this alternating behavior offers modest improvements over conventional recommenders (which exhibit monotonic rank decay), it fails to fundamentally prevent collapse as model depth increases. The paper posits that the rigid, non-parameteric nature of the token mixing and the standard activation functions in P-FFNs are the primary causes of this spectral instability.

2. Methodology: RankElastor

To address these limitations, the authors propose RankElastor, a novel deep recommender architecture designed to produce spectrum-robust representations with provable collapse mitigation. The core philosophy is "Expand More, Shrink Less," achieved through two key architectural innovations:

A. Parameterized Full Token Mixing

In RankMixer, token mixing is implemented via a non-parameteric block-transpose operation. The authors generalize this by introducing a learnable mixing operator that reduces the grid scale to the finest resolution ( $d=1$ ).

Mechanism: Instead of a fixed permutation, RankElastor employs a dense linear transformation over the flattened representation: $vec(M^\top) = LN((W + I)vec(X^\top))$ , where $W$ is a learnable mixing matrix.
Theoretical Basis: Theorem 3.1 demonstrates that coarse-grained block mixing (where $d^* > 1$ ) is strictly less expressive than full mixing ( $d^* = 1$ ). The block-Kronecker constraint in standard mixing prevents the model from expressing fine-grained, high-rank coordinate interactions, particularly when input representations are already spectrally collapsed. Full mixing removes this constraint, enabling richer interactions.

B. GLU-Improved P-FFNs

The authors replace the standard GELU-based P-FFNs in RankMixer with GLU-style gated feedforward modules.

Mechanism: The new module utilizes gated activations: $Z_t = (GELU(M_t W_1) \odot (M_t W_2))W_3 + M_t W_r$ .
Theoretical Basis: Theorem 3.2 proves that GLU-improved P-FFNs provide stronger effective rank recovery capabilities. Unlike standard FFNs which can deterministically or probabilistically contract effective rank (especially on low-rank inputs), the multiplicative nature of the GLU gate induces degree-2 interactions. This allows the module to statistically maintain spectrum robustness and induce an additive increase in effective rank ( $\delta > 0$ ), provided the hidden width satisfies specific scaling conditions.

3. Key Contributions

Analysis of RankMixer: The paper provides both empirical evidence and theoretical justification showing that RankMixer's core modules (token mixing and P-FFNs) jointly constrain spectral robustness, leading to incomplete mitigation of embedding collapse.
RankElastor Architecture: The proposal of a new architecture that integrates Parameterized Full Mixing and GLU-improved P-FFNs. The authors provide theoretical guarantees on the spectral robustness of these components, showing they enable more expressive transformations and prevent rank contraction.
Empirical Validation: Extensive experiments on industrial-scale benchmarks demonstrate that RankElastor consistently outperforms strong baselines (including RankMixer, DCNv2, and xDeepFM) in CTR prediction while maintaining higher effective rank and exhibiting superior parameter scaling behavior.

4. Experimental Results

The authors evaluated RankElastor on two large-scale industrial datasets: Criteo and Avazu.

Prediction Performance: RankElastor achieved state-of-the-art results, improving AUC by over 0.001 compared to the strongest baseline (RankMixer) on both datasets. This gain is considered statistically significant in industrial CTR benchmarks.
Embedding Collapse Mitigation:
- Effective Rank: RankElastor consistently maintained a higher average effective rank across all layers compared to RankMixer.
- Dynamics: While RankMixer showed a damped oscillatory trajectory with significant contraction at FFN layers, RankElastor exhibited stronger expansion during mixing and substantially milder contraction during FFN stages, leading to stable spectral evolution.
Scaling Behavior: Scaling-law experiments (varying depth and width) revealed that RankElastor scales more effectively than RankMixer. Joint scaling of depth and width yielded larger performance gains, confirming its potential as a scalable paradigm.
Generalization: The model was also tested on behavior sequence modeling tasks (KuaiVideo and TaobaoAd), where it again outperformed baselines, suggesting the approach generalizes beyond standard CTR prediction.
Efficiency: The proposed architecture introduces a modest 10–15% increase in runtime compared to RankMixer due to the parameterized mixing, while maintaining comparable GPU memory usage.

5. Significance and Claims

The paper claims that RankElastor offers a principled direction for refining token-transformation-based recommenders. By shifting from rigid, non-parameteric operations to learnable, fine-grained mixing and gated activations, the architecture successfully decouples the trade-off between model depth and representation collapse.

The authors emphasize that their work provides a spectral perspective on representation collapse, identifying that the "damped oscillatory trajectory" is an inherent property of the original RankMixer design. RankElastor is presented not just as an incremental improvement, but as a robust and scalable paradigm that unlocks the potential of dense scaling in recommendation systems by ensuring that expanded representation spaces are effectively utilized rather than collapsed.

The paper concludes by noting that while the findings are specific to this class of architectures, the insights into spectral dynamics and the "Expand More, Shrink Less" principle offer a foundation for future research into more expressive token-mixing mechanisms and spectrum-preserving nonlinear modules.