Technical Summary: Heterogeneous Ranking in Industrial-Scale Recommender Systems

1. Problem Statement

The paper addresses the challenge of unified ranking in Google Discover, a system that aggregates highly heterogeneous content from the decentralized open web (e.g., web articles, long/short-form videos, user-generated content, sports cards, and AI summaries) into a single personalized feed. Unlike closed-ecosystem platforms (e.g., music or video-only apps) where content modalities and interaction patterns are uniform, Google Discover faces two primary dimensions of heterogeneity:

• Data Heterogeneity: Content lacks uniform features, leading to variations in feature densities, metadata structures, and asymmetric signal availability.
• Interaction Heterogeneity: User engagement patterns and intents vary significantly by content type. For instance, while "likes" and "dismissals" are universal, articles primarily drive clicks, whereas UGC and sports cards drive long dwell times.

Key Challenges:

1. Distribution Mismatch: A model optimized for global metrics (e.g., overall CTR) may disproportionately favor dominant, low-quality content types (e.g., click-bait articles) over high-value minority types (e.g., videos), degrading the user experience.
2. Negative Transfer (Seesaw Phenomenon): Jointly optimizing conflicting objectives (e.g., maximizing clicks vs. minimizing dismissals) in a shared representation often causes improvements in one task to negatively impact others.
3. Minority-Type Collapse: Shared architectures tend to favor dominant content types with richer features or higher traffic, causing underrepresented content types to be "drowned out."

The existing production baseline uses a shared Multi-Layer Perceptron (MLP) backbone with 11 task-specific heads, which forces a single latent representation to serve all content types, failing to adequately address these heterogeneities.

2. Methodology

The authors propose an end-to-end workflow comprising a new model architecture (HA-MoE), an observability framework (LENS), and a specialized evaluation metric (DL-AUC).

2.1 HA-MoE: Heterogeneity-Adaptive Multi-gated Mixture-of-Experts

To address the challenges without violating strict industrial constraints (model size, training speed, and serving latency), the authors introduce HA-MoE. This architecture integrates explicit heterogeneity signals ($h$) into both the gating networks and expert representations.

• Architecture: It utilizes a dense, expert-based multi-task learning framework with $N$ shared experts (Feed-Forward Networks).
• Heterogeneity-Aware Gating (HA-Gating): The gating mechanism explicitly incorporates heterogeneity signals ($h$) alongside dense features ($x$) to determine expert utilization for each task. This allows the model to steer specific experts toward specific content segments, protecting minority segments from being overshadowed.
  $$g_t(x, h) = \text{Softmax}(W_t \Phi(x, h) + b_t)$$
• Heterogeneity-Driven Linear Modulation (HDLM): To decouple expert capacity from distribution-specific biases, the output of each expert is dynamically adapted using scale ($\gamma$) and shift ($\beta$) vectors derived from $h$. This allows the network to "switch contexts" (e.g., shifting focus from visual patterns for videos to creator affinity for followed creators) without massive parameter expansion.
  $$\tilde{E}_n(x, h) = \gamma_n(h) \odot E_n(x) + \beta_n(h)$$
• Optimization: The model uses a combined loss function balancing pointwise Binary Cross-Entropy (BCE) and pairwise RankNet loss to calibrate absolute probabilities while preserving relative ordering.

2.2 LENS: Latent Expert Network Specialization

To mitigate the "black box" nature of unified models, the authors introduce LENS, a lightweight observability framework:

• Activation Slicing: Visualizes the model's internal division of labor by slicing gating activations into content-by-expert matrices, revealing how different experts specialize in different content types.
• Permutation-Invariant Expert Matching (PIEM): Since MoE experts are permutation-invariant, direct parameter comparison is intractable. PIEM characterizes each expert by a behavioral profile vector (relative contribution across tasks) and uses the Hungarian algorithm to find the optimal 1-to-1 matching between model snapshots. It calculates a score based on Jensen-Shannon Divergence (JSD) to track functional stability and detect "functional collapse" (loss of specialization) during continuous retraining.

2.3 DL-AUC: Dual-Level AUC

Standard global AUC can mask structural failures where a model inflates scores for majority segments while suppressing minority ones. The authors propose DL-AUC to balance global utility with cross-segment correctness:
$$\text{DL-AUC} = \lambda \cdot \text{Micro-AUC} + (1 - \lambda) \cdot \text{Macro-xAUC}$$

• Micro-AUC: Measures global ranking quality under the empirical traffic distribution.
• Macro-xAUC: The unweighted average of xAUC across all pairs of content segments. This ensures that a positive item from a minority segment (e.g., Video) is correctly ranked higher than a negative item from a majority segment (e.g., Web Article), preventing majority class domination.

3. Key Contributions

1. HA-MoE Architecture: A production-ready, heterogeneity-adaptive MoE design that explicitly injects content context into gating and expert layers, enabling effective specialization without significant operational overhead (<5% model size increase, <0.5% latency increase).
2. LENS Framework: A novel observability tool that "unboxes" internal model dynamics, allowing engineers to interpret expert specialization and quantitatively track functional heterogeneity across training cycles using PIEM.
3. DL-AUC Metric: A heterogeneity-aware evaluation metric that prevents the deployment of models that achieve global gains at the expense of ecosystem health (cross-segment correctness).

4. Results

Offline Evaluation

• Performance: On a large-scale industrial dataset (10M examples), HA-MoE achieved the highest aggregated DL-AUC (0.691 for pInterest, 0.949 for pDisinterest), outperforming both the Shared MLP baseline and Standard MMoE.
• Ablation: Both HA-Gating and HDLM components contributed to the gains. The full model successfully reversed the negative transfer seen in baselines, improving negative engagement predictions (pDisinterest) while maintaining positive gains.
• Cross-Type Correction: HA-MoE significantly narrowed the ranking gap between Web Articles and Videos. The gap in cross-type xAUC (Vid+ vs. Web−) improved from 0.141 (Shared MLP) to 0.060 (HA-MoE), demonstrating the model's ability to prevent low-quality articles from suppressing high-quality videos.

Online A/B Testing

Deployed on 1% of live traffic in Google Discover (hundreds of millions of DAU), the HA-MoE model showed statistically significant lifts:

• Feed Activity: +0.22% DAU and +0.48% Viewed Impressions.
• Exploration: +0.34% Scroll Depth, +0.36% Diverse Feed Rate, and +0.54% Diverse Engagement Rate.
• Stability: No regressions were observed in core operational metrics. The results validated that improved heterogeneous ranking translates to tangible ecosystem health benefits.

5. Significance and Claims

The paper claims to present a production case study that validates the effectiveness of explicitly modeling heterogeneity in unified ranking systems. Its significance lies in:

• Industrial Scalability: Demonstrating that complex heterogeneity adaptation can be achieved within strict production constraints (latency, storage, training speed), moving beyond theoretical proposals to deployed reality.
• Ecosystem Health: Proving that balancing global utility with cross-segment correctness (via DL-AUC and HA-MoE) prevents the "majority bias" that typically degrades the user experience in open-web feeds.
• Operational Transparency: Establishing LENS as a critical tool for maintaining structural stability in continuous training pipelines, ensuring that model updates do not inadvertently collapse functional specialization.

The authors conclude that while managing heterogeneity at scale remains an evolving challenge, their approach provides a robust design pattern for industrial recommender systems, bridging the gap between unified modeling and the diverse realities of the open web. Future work is noted to focus on more expressive gating, sparse routing, and extending heterogeneity modeling to user-level context and listwise optimization.