Technical Summary: Constructive Multi-Sequence Learning (CMSL) for Recommendation Systems

1. Problem Statement: Context Pollution in Sequential Recommendation

While sequence learning has surpassed traditional Deep Learning Recommendation Models (DLRM) by capturing temporal nuances, current state-of-the-art architectures often rely on a limiting analogy: treating user history as a monolithic chronological sequence, similar to a sentence in a Large Language Model (LLM).

The authors identify a fundamental divergence between natural language and recommendation data:

• Lack of Structural Unity: Unlike text, which follows linear, logical syntactic rules, user history is a "noisy mosaic" of fragmented, diverse interests, exploratory clicks, and accidental interactions.
• Context Pollution: In single-sequence modeling, unrelated behaviors compete for the same attention budget. This "noisy" signal dilutes the model's focus, forcing it to compute dependencies between semantically distinct intents (e.g., professional learning vs. casual entertainment). This leads to spurious correlations, representation bottlenecks, and suboptimal predictions.

The paper argues that recommendation systems should shift from passively "modeling a sequence" to actively "constructing context," addressing the under-explored issue of context pollution in sequential recommendation.

2. Methodology: Constructive Multi-Sequence Learning (CMSL)

CMSL proposes a paradigm shift from passive sequence ingestion to active "context engineering." Instead of feeding raw history as a single prompt, CMSL constructs multiple coherent latent sequences in the latent space to disentangle user intents.

2.1 Feature Preprocessing

• Non-Sequence Features: Dense features (e.g., age, price) are normalized and projected; sparse features (e.g., country, category) are embedded.
• Sequence Features: User interaction histories (clicks, likes, comments) are grouped by shared attributes (e.g., same product surface) and compressed into a single sequence using a gated summation mechanism. This reduces computational overhead while preserving interaction information.

2.2 CMSL Architecture

The core of CMSL consists of $N$ stacked blocks, each containing four key components:

1. Multi-Sequence Construction:

   • Intent Disentanglement: Non-sequence features are projected into $K$ context-specific sets using $K$ projection heads, effectively disentangling user intents into different contexts.
   • Sequence Construction: A learnable Sequence Construction Module uses cross-attention. The current sequence representation serves as the query ($Q$), while the projected context features serve as keys ($K$) and values ($V$). This generates $K$ distinct latent sequences ($\tilde{X}^{(l,k)}$), each focusing on a different aspect of the user's context.

2. Self-Attention (Linear Approximation):

   • Each of the $K$ constructed sequences undergoes self-attention independently to capture internal temporal correlations without cross-intent interference.
   • To address the quadratic complexity of standard attention and the $K$-fold increase in computation, CMSL employs a linear-time attention mechanism.
   • This mechanism approximates the HSTU attention (widely used at Meta) using a Taylor series expansion of the SiLU activation function. It decomposes the operation into a linear attention term ($\phi(Q)\phi(K)^\top V$) and a sparse term ($AV$) that captures large entries, significantly reducing computational cost.

3. Sequence Summarization:

   • After self-attention, Pooling by Multi-head Attention (PMA) is used to summarize each sequence into a fixed-length vector.
   • This step extracts global signals relevant to the user's current state (using non-sequence features as queries) and suppresses redundant historical data, preventing noise accumulation across layers.

4. Sequence Compression:

   • To prevent the sequence count from exploding across layers, the $K$ summarized sequences are concatenated and projected back into a single latent sequence via an MLP. This allows for deep hierarchical learning while maintaining a constant sequence count per layer.

2.3 Final Fusion

The final representation is formed by concatenating the summarized vectors from the final layer and passed to task-specific heads for Click-Through Rate (CTR) and Conversion Rate (CVR) predictions.

3. Key Contributions

The paper outlines four primary contributions:

1. New Perspective: Identifies "Context Pollution" as a critical failure mode in sequential recommendation and proposes a multi-sequence construction paradigm to mitigate it.
2. Multi-Sequence Construction Module: A learnable module that transforms raw history into multiple latent streams using cross-attention for intent-aware, adaptive context curation.
3. Scalable Linear Attention: An efficient linear-time attention mechanism that reduces the computational cost of modeling multiple sequences, enabling scale.
4. Industrial-Scale Validation: Successful deployment and validation across four major surfaces at Meta, handling billions of daily active users.

4. Experimental Results

CMSL was evaluated on Ranking and Retrieval tasks across Meta's production environment.

• Ranking Evaluation:

  • Metric: Normalized Entropy (NE), where lower values indicate better performance.
  • Setup: Evaluated on four large-scale surfaces. Surface 1 used a direct training approach, while Surfaces 2–4 utilized a two-stage architecture (asynchronous upstream user model) to manage latency.
  • Findings: CMSL demonstrated statistically significant improvements. On Surface 1, NE improvements ranged from -0.31% to -0.62% for "Like" and "Comment" tasks. On Surfaces 2–4, integrating CMSL embeddings yielded NE gains of -0.06% to -0.13% across CTR and CVR tasks.

• Retrieval Evaluation:

  • Setup: Online A/B testing on Surface 5.
  • Findings: CMSL user embeddings resulted in statistically significant improvements across all four measured engagement metrics, with gains ranging from +0.092% to +0.171%.

5. Significance and Claims

The paper claims that CMSL represents a shift from treating user history as a "raw prompt" to be read, to a "context" to be constructed. By actively disentangling user history into coherent thematic strands, CMSL mitigates the semantic interference inherent in heterogeneous user data.

The authors emphasize that this approach aligns with the "context engineering" philosophy of LLMs but adapts it for recommendation systems where item IDs lack explicit semantics, necessitating an implicit, learnable approach. The framework is presented as a practical solution for production-scale systems, offering consistent gains over strong sequential baselines while remaining computationally feasible through linear attention and sequence compression.