Technical Summary: Bridging the Information Gap with SemRaD

1. Problem Definition

The paper addresses the new-user cold-start problem in e-commerce, specifically the challenge of predicting User Lifetime Value (LTV) and Conversion Rate (CVR) for users with sparse interaction histories. Standard sequence models (e.g., DIN, SASRec) rely on rich historical data; when user traces contain only a few events, predictions degenerate toward population averages, leading to inefficient marketing and missed retention opportunities.

The authors identify two existing approaches and their specific limitations:

1. LLM-based Semantic Augmentation: While Large Language Models (LLMs) can densify sparse histories by inferring user intent, they typically generate unstructured, free-form rationales. In production, these are noisy, difficult to validate, and inconsistent across users or market shifts, making them hard to operationalize.
2. Learning Using Privileged Information (LUPI): This approach uses a "teacher" model with access to post-conversion signals (privileged information) to train a "student" model that only sees pre-conversion data. However, naive distillation often fails because the information gap between the teacher and student is too large and heterogeneous across users. A single shared distillation pathway tends to average over these regimes, failing to transfer knowledge effectively to the specific users who need it most.

2. Methodology: SemRaD Framework

The authors propose SemRaD (Semantic Reasoning-aware Distillation), a framework designed to bridge the information gap through two complementary components: a Structured Semantic Reasoning Pipeline and a Hindsight-Aware Distillation Network.

A. Structured Semantic Reasoning Pipeline

Instead of generating free-form text, SemRaD employs a discover–curate–audit workflow to create a fixed, interpretable schema of behavioral dimensions (e.g., Temporal Resolution, Transaction Magnitude).

• Schema Construction: LLMs propose candidate dimensions, which are deduplicated and consolidated by a second LLM call, then reviewed by domain experts to ensure distinctness and predictive relevance.
• Semantic Profiling:
  • Pre-conversion ($Z_{pre}$): The LLM analyzes the sparse pre-conversion log to generate a Densified Semantic Profile consisting of structured rationales for each dimension. This is the only LLM-derived signal used by the deployed student.
  • Post-conversion ($Z_{post}$): During training, the LLM analyzes the privileged post-conversion log to generate a future profile.
  • Hindsight Fusion ($Z_{fused}$): A fusion prompt reconciles the pre- and post-conversion reasoning. It identifies which pre-conversion signals were truly predictive and resolves contradictions, producing a Hindsight Distillation Target. This target provides the teacher with self-consistent supervision that the student can learn from, rather than raw, potentially contradictory signals.

B. Hindsight-Aware Distillation Network

The framework uses a Simultaneous Distillation paradigm with a Student (online) and Teacher (training only).

• Semantic-Gated Encoder: To handle user heterogeneity (where different dimensions matter for different users), the model uses a Semantic-Gated Encoder. It encodes each semantic rationale with a frozen text embedder and computes per-user importance weights via a gating mechanism. This allows the model to focus on the most informative dimensions for each specific user.
• Distillation Experts: To address the heterogeneous information gap, the framework replaces a single shared distillation pathway with Distillation Experts. A Gumbel-softmax router, conditioned on the teacher's representation, selects among multiple expert MLPs to reconstruct the teacher's target from the student's input. This allows the model to adapt the transfer strategy based on the specific user's data regime.
• Training Objective: The total loss combines the task-specific supervised loss (Huber for LTV, BCE for CVR), soft-target knowledge distillation (KD), and the expert-routed reconstruction loss.

3. Key Contributions

1. Structured Semantic Reasoning Pipeline: Replaces unstructured LLM rationales with a schema-guided, discover-curate-audit workflow. This produces a Densified Semantic Profile for inference and a Hindsight Distillation Target for training, ensuring consistency and operationalizability.
2. Hindsight-Aware Distillation Network: Introduces a distillation mechanism that bridges the teacher-student gap via the reconciled hindsight target and handles per-user variability through Distillation Experts, avoiding the brittleness of single-pathway distillation.
3. Production Deployment & Generalization: Demonstrates that SemRaD generalizes across tasks (LTV and CVR), student backbones (Transformer, DIN, Avg-Pooling), and profiler LLMs. It validates the approach in a large-scale industrial setting.

4. Experimental Results

Experiments were conducted on a large-scale industrial dataset (120k users) and validated via a four-week online A/B test at Keeta.

• Offline Performance: Compared to a production-grade base model, SemRaD achieved:
  • +1.9% improvement in LTV (Gini Coefficient).
  • +1.0% improvement in CVR (AUROC).
  • Ablation studies confirmed that both the structured semantic profiling and the hindsight-aware distillation (including experts) are necessary for peak performance.
• Data Efficiency: SemRaD matched the production system's LTV performance using only 9% of the training data while improving CVR by 0.8%, highlighting the signal efficiency of the semantic profiles.
• Online A/B Test: In a live deployment (5% traffic, ~10k users), SemRaD showed:
  • +1.0% relative gain in LTV.
  • +0.43% relative gain in CVR.
  • Serving latency increased slightly (P50: +7.6%, P99: +34%) due to the gated MLP, but no LLM calls were required at inference time (profiles are cached).
• Mechanism Analysis:
  • Gate Weights: The learned gating weights aligned with expert intuition (e.g., Navigation Intent Specificity dominated), validating the schema utility.
  • Distillation Experts: The router naturally clustered users into distinct regimes (e.g., converters vs. non-converters) without explicit supervision.
  • Information Gap: SemRaD significantly reduced the prediction disagreement between the teacher and student for cold-start users compared to naive distillation, confirming that hindsight fusion makes privileged supervision more reachable.

5. Significance and Claims

The paper claims that SemRaD effectively addresses the dual challenges of unstructured semantic noise and heterogeneous information gaps in cold-start prediction. By converting LLM reasoning into a structured schema and reconciling privileged signals through hindsight fusion, the framework enables robust knowledge transfer from a privileged teacher to a sparse-data student.

The authors emphasize that the framework is model-agnostic and has been successfully adopted for other production tasks at Keeta, including CTR prediction, user simulation, and silent-user re-engagement. The work demonstrates that combining structured semantic reasoning with advanced distillation techniques can yield significant, measurable business value while maintaining data efficiency and operational feasibility.