Versioned Late Materialization for Ultra-Long Sequence Training in Recommendation Systems at Scale

1. Problem Statement: The "Fat Row" Bottleneck

Modern Deep Learning Recommendation Models (DLRMs) follow scaling laws where model quality improves monotonically with longer User Interaction History (UIH) sequences (moving from $10^4$ to $10^5+$ events). However, the industry-standard approach to handling this data creates a critical infrastructure bottleneck known as the "Fat Row" problem.

• Pre-Materialization Redundancy: Current systems physically duplicate (pre-materialize) the entire UIH sequence into every single training example to ensure Online-to-Offline (O2O) consistency.
• K-Fold Amplification: If a user has $N$ days of history and generates $K$ ranking requests within a lookback window, the same historical data is stored $K$ times. This leads to massive storage and I/O redundancy.
• The Resource Wall: As sequence lengths increase, the resource consumption of data supporting services (storage, ingestion, preprocessing) exceeds the capacity of GPU training clusters. This creates a "diminishing returns" trap where infrastructure costs cap the ability to scale sequence lengths, effectively preventing the adoption of next-generation architectures like HSTU and ULTRA-HSTU.
• Multi-Tenant Penalty: In shared union datasets serving multiple model tenants (e.g., retrieval vs. ranking models with different sequence length needs), the "Fat Row" paradigm forces all models to read the maximum sequence length, causing unnecessary I/O amplification for shorter-sequence models.

2. Methodology: Versioned Late Materialization

The authors propose a paradigm shift from storage-heavy pre-materialization to a compute-centric late materialization approach. The core insight is that UIH is an append-only, temporally ordered, immutable sequence. Therefore, the exact state of a user's history at any inference time $t$ can be reconstructed deterministically without storing physical snapshots.

A. The Protocol

The system employs a bifurcated protocol to ensure O2O consistency and prevent future leakage:

1. Inference-Time Snapshotting:
   • Instead of logging the full UIH, the system logs only lightweight version metadata (temporal boundaries start_ts, end_ts, sequence length, and a checksum) alongside the mutable (recent) UIH portion.
   • This decouples logging write-amplification from the total sequence length ($O(1)$ metadata vs. $O(seq\_length)$ data).
2. Training-Time Reconstruction:
   • During training data preprocessing, the system performs a "Time-Travel" reconstruction.
   • It retrieves the recent mutable UIH from the training example and issues a bounded range scan against a normalized, immutable UIH store using the logged temporal boundaries.
   • The two components are concatenated to recreate the exact inference-time state.

B. Storage Architecture

• Hybrid Storage:
  • Mutable UIH: A real-time store for recent events (seconds-level freshness), accessed only during inference.
  • Immutable UIH: A normalized, read-optimized store for long-term history. It uses a single-level layout with pre-sorted, chronologically ordered files to ensure purely sequential I/O during range scans.
• Multi-Dimensional Projection: The immutable store organizes data by user_id, feature_group, and subsequence_timestamp. This allows the system to push down projections at the storage layer:
  • Sequence Length Projection: Shorter models only scan the required number of temporal stripes.
  • Feature Group Projection: Models only read relevant feature columns.
• Trait-Aware Encoding: Uses columnar encoding (delta-encoding, bitmaps, dictionary compression) to handle sparse vs. dense features, enabling selective decoding at the byte level.

C. Training Pipeline Optimization

To prevent the reconstruction process from stalling GPU training (GPU starvation), the system implements:

• Disaggregated Data Preprocessing (DPP): Offloads heavy I/O and decoding to a separate, elastically scalable cluster of preprocessing workers.
• Pipelined I/O Prefetching: Overlaps the reading of the next training batch with the current batch's sequence lookup to mask latency.
• Data-Affinity Sharding: Ensures that training examples and their corresponding UIH data reside on the same storage shard (via identical hash partitioning), minimizing network fanout and maximizing cache hit rates for batch jobs.

3. Key Contributions

1. Formalization of the Fat Row Problem: Quantitative analysis demonstrating that pre-materialized UIH redundancy drives data infrastructure usage beyond GPU training capacity, capping model scaling.
2. Versioned Late Materialization Protocol: A unified solution for both streaming and batch training that guarantees O2O consistency and prevents future leakage using lightweight version metadata and temporal predicates.
3. Production-Grade Architecture: A system combining read-optimized immutable storage, multi-tenant projection pushdown, and disaggregated preprocessing to mask reconstruction latency, keeping training throughput GPU-bound.
4. Scale Deployment: Successful deployment on Meta's production DLRMs, enabling aggressive sequence length scaling (up to $10^5$ events) while reducing infrastructure resource usage.

4. Results

The system was evaluated across three model tenants sharing a union dataset and two production recommendation platforms.

• System Efficiency:

  • Write Bandwidth: Reduced primary write bandwidth by 46.2% by eliminating $K$-fold redundancy.
  • Read Bandwidth: Reduced per-job read bandwidth by 47–70%. Shorter-sequence models (B and C) saw further gains (up to 70% reduction) due to projection pushdown.
  • Latency: Despite adding sequence lookups, the system maintained low latency. Model A (long sequence) saw only a 9.7% increase in data loading latency, absorbed by elastic DPP scaling. Models B and C saw 26–36% latency improvements due to projection pushdown.
  • Throughput: The immutable storage delivered 3.4× higher read throughput per unit of host resource compared to the baseline append-only storage.

• Model Performance:

  • Scaling Breakthrough: The system broke through the "Fat Row Wall" (where data costs exceed GPU costs at ~4K sequence length).
  • Quality Gains:
    • Platform A: Achieved an additional 1.2% cumulative Normalized Entropy (NE) gain by scaling from 4K to 64K events (total >5% gain).
    • Platform B: Achieved 0.65% NE gain scaling from 4K to 10K events.
  • Online Impact: A/B tests confirmed significant improvements in engagement metrics (Topline, Consumption, Explicit Engagement) when scaling sequence lengths, gains that were previously impossible under the Fat Row paradigm.

5. Significance

This paper establishes that data infrastructure is a first-class scaling lever for recommendation systems, complementary to model architecture advances. By treating UIH as an immutable, versioned sequence rather than a static snapshot, the authors eliminated the storage and I/O walls that previously prevented the industry from training on ultra-long sequences. This paradigm enables the practical deployment of generative recommendation frameworks and trillion-parameter sequential transducers, unlocking significant model quality improvements that were previously cost-prohibitive.