Technical Summary: Kunlun – Establishing Scaling Laws for Massive-Scale Recommendation Systems

1. Problem Formulation

Modern recommendation systems, particularly in large-scale advertising, require the joint modeling of sequential user behaviors (e.g., click history) and non-sequential context features (e.g., user demographics, ad metadata). While scaling laws—predictable mathematical relationships between computational investment and model performance—are well-established for Large Language Models (LLMs), they remain an open challenge for recommendation systems that handle this heterogeneous feature mix.

The authors identify poor scaling efficiency as the primary barrier to establishing these laws. Scaling efficiency is defined as the rate of performance improvement per unit of computational investment. The paper argues that current state-of-the-art systems fail to achieve predictable power-law scaling due to two critical bottlenecks:

1. Inefficient Modules: Existing architectures suffer from low Model FLOPs Utilization (MFU), often ranging between 3–15% compared to 40–60% in LLMs. This is caused by heterogeneous feature spaces leading to small embedding dimensions, irregular tensor shapes, and memory-bound operations.
2. Inefficient Resource Allocation: Naively scaling all components equally leads to diminishing returns, as different layers and event types (e.g., clicks vs. impressions) benefit from distinct computational patterns.

2. Methodology: The Kunlun Architecture

To address these challenges, the authors propose Kunlun, a unified architecture designed through systematic model-efficiency codesign at two levels: low-level module optimization and high-level computation reallocation.

2.1 Architecture Overview

Kunlun employs a multi-layer architecture where each layer processes both sequence and non-sequential features through two main components:

• Kunlun Transformer Block: Handles context-aware sequence modeling using Multi-Head Self-Attention (MHA) and a Generalized Dot-Product Attention (GDPA)-enhanced Personalized Feed-Forward Network (PFFN).
• Kunlun Interaction Block: Facilitates bidirectional information exchange via Weight Generation, Hierarchical Seed Pooling (HSP), and a Global Interaction module.

2.2 Low-Level Module Optimizations

These optimizations target computational efficiency to improve MFU:

• Generalized Dot-Product Attention (GDPA): The authors reformulate the PFFN as a multi-head attention operator. Instead of separate matrix multiplications that create memory bottlenecks, GDPA fuses the transformation into a single kernel (similar to FlashAttention). Here, sequence tokens act as queries, while weights derived from non-sequential context act as keys and values. This enables efficient GPU utilization and allows for stable PFFN stacking.
• Hierarchical Seed Pooling (HSP): To summarize long sequences ($T > 1000$) efficiently, HSP replaces random learnable queries (used in standard Pooling by Multihead Attention) with a hierarchical process. It first learns seed-level representations and then compresses them using SumKronLinear, a Kronecker product-based layer. This achieves parameter efficiency (reducing parameters by ~14× in typical configurations) while maintaining cross-dimensional expressiveness.
• Sliding Window Attention: Recognizing the strong locality bias in recommendation sequences (recent interactions are more predictive), the model restricts self-attention to local windows. This reduces complexity from $O(T^2)$ to $O(Tw)$, significantly improving throughput.

2.3 High-Level Computation Reallocation

These strategies optimize resource distribution across the model:

• Computation Skip (CompSkip): Inspired by layer redundancy in LLMs, Kunlun implements an "every-other-layer" pattern. Even layers skip self-attention to compute fresh HSP summaries, while odd layers skip HSP and PFFN to compute self-attention. This reduces FLOPs by ~43% while maintaining model quality.
• Event-Level Personalization: Different event types (e.g., clicks vs. impressions) are assigned distinct hyperparameters (model dimension, number of heads, summarization tokens, layer depth, and window size). High-value events receive larger configurations, while lower-value events use smaller ones, ensuring resources are allocated proportionally to signal importance.
• Mixture of Wukong Experts: The Global Interaction module employs a Mixture-of-Experts (MoE) approach where multiple "Wukong" modules (feature interaction models) operate in parallel. This enables horizontal scaling by distributing computation across experts within a single layer.

3. Key Contributions

The paper outlines four primary contributions:

1. Problem Formulation: Formalizing the scaling efficiency challenge in massive-scale recommendation systems, identifying inefficient modules and resource allocation as the root causes of unpredictable scaling.
2. Architecture Design: Introducing Kunlun, which integrates low-level optimizations (GDPA, HSP, Sliding Window) and high-level innovations (CompSkip, Event-level Personalization) to unify sequence and non-sequential modeling.
3. Scaling Laws: Demonstrating the first predictable power-law scaling behavior for joint sequence-nonsequence modeling, achieving a 2× improvement in scaling efficiency over state-of-the-art methods.
4. Deployment Impact: Successful deployment in major Meta Ads models, resulting in measurable production improvements.

4. Experimental Results

The authors evaluated Kunlun on large-scale internal Meta Ads datasets (70B+ samples) using NVIDIA B200 GPUs.

• Scaling Efficiency: Kunlun achieved an MFU increase from 17% to 37%. In terms of scaling efficiency (the slope of performance improvement vs. compute), Kunlun demonstrated a 2× improvement over the current state-of-the-art (InterFormer).
• Performance Gains: Across computational scales (6 to 180 GFLOPs), Kunlun consistently outperformed baselines. At the largest scale (180 GFLOPs), Kunlun achieved 0.79% NE (Normalized Entropy) gain over the baseline, compared to 0.50% for InterFormer.
• Ablation Studies:
  • Removing GDPA reduced MFU to 34% and degraded NE by 0.04%.
  • Removing HSP (reverting to PMA) improved QPS by 8.8% but degraded NE by 0.08%.
  • Removing CompSkip increased FLOPs by 43.1% with no NE gain, validating the efficiency of the skip strategy.
• Production Impact: Upon deployment in Meta Ads models, Kunlun delivered a 1.2% improvement in topline metrics, demonstrating significant online impact and superior ROI at scale.

5. Significance and Claims

The paper claims that Kunlun establishes a principled framework for scaling recommendation systems to unprecedented scales while maintaining computational efficiency. By systematically addressing both algorithmic effectiveness and hardware utilization, the authors demonstrate that predictable scaling laws are achievable for complex, heterogeneous recommendation tasks.

The work asserts that the combination of low-level module efficiency (via GDPA and HSP) and high-level adaptive resource allocation (via CompSkip and Event-level Personalization) is necessary to unlock the full potential of massive-scale models. This enables reliable extrapolation for resource planning and architectural decisions, moving recommendation systems from heuristic scaling to predictable, law-governed growth. The successful deployment at Meta validates that these theoretical gains translate directly to production value.