Technical Summary: Mind the Gap: Bridging Behavioral Silos with LLMs in Multi-Vertical Recommendations

1. Problem Statement

In multi-vertical e-commerce ecosystems like DoorDash, expanding into new domains (e.g., grocery, retail, beauty) creates a significant personalization challenge: the "cold start" problem. Users often lack engagement history in these emerging verticals, making it difficult to generate relevant recommendations. Conversely, these platforms possess rich behavioral data from established, high-traffic verticals (e.g., restaurant ordering). The core challenge addressed by this work is how to effectively transfer latent knowledge from data-rich verticals to data-sparse ones to enhance recommendation quality without relying on direct historical interactions in the new domain.

2. Methodology

The authors propose a novel framework that utilizes Large Language Models (LLMs) to perform semantic feature engineering, synthesizing structured user affinity features from unstructured behavioral data.

2.1 Hierarchical Retrieval-Augmented Generation (RAG)

The system employs a hierarchical RAG pipeline to map unstructured user signals (restaurant order histories and search queries) to a four-level internal product taxonomy (L1–L4).

• Cascaded Inference: The process operates via a cascaded inference strategy. The model first identifies broad, top-level (L1, L2) taxonomic affinities. These high-confidence classifications then constrain the generative search space for subsequent, more granular retrievals at lower levels (e.g., L3). This iterative refinement prevents the generation of plausible but incorrect subcategories.
• Feature Focus: For the production Multi-Task Learning (MTL) ranker, the system specifically targets L2 and L3 affinities, as L1 is often too general and L4 suffers from excessive sparsity.

2.2 Generative Feature Synthesis

The framework generates two distinct classes of features to augment the user representation:

1. Long-Term Cross-Vertical Affinity Profile: Captures latent, historical preferences inferred from cumulative restaurant order history.
2. Short-Term Intent Profile: Models recent, transient interests derived from on-platform search activity.

2.3 Prompt Engineering and Control

To ensure high-fidelity and deterministic outputs, the authors implemented strict prompt engineering controls:

• Contextual Input: Prompts concatenate historical restaurant names and ordered items chronologically, prioritizing recent behavior, alongside search queries.
• Confidence Filtering: The model is instructed to assign a confidence score to each generated affinity. Only taxonomies exceeding a confidence threshold of 0.8 are included. This acts as a self-correction mechanism to filter spurious associations.
• Output Constraints: The model is restricted to outputting exact taxonomy spellings from a provided list, preventing the invention of non-existent categories.

2.4 Integration into Production MTL

The LLM-derived features are integrated into the existing production Multi-Task Learning (MTL) ranking model:

• Feature Augmentation: The new sparse features ($u_{LLM}$), representing user affinities, are concatenated with existing user engagement vectors ($u_{eng}$) and item feature vectors ($i_{eng}$).
• Embedding Strategy: Variable-length categorical features (lists of taxonomy IDs) are mapped to dense vectors via a shared embedding table and aggregated using mean pooling to handle jagged input tensors.
• Model Architecture: The augmented feature vector feeds into a shared MLP trunk followed by task-specific prediction heads, jointly optimizing for objectives such as click-through rate, add-to-cart, and purchase.

2.5 Optimization

• Model Selection: A cost-performance analysis identified GPT-4o-mini as the optimal model, delivering comparable quality to GPT-4o at a substantially lower computational cost.
• Cost Reduction: Implementation of prompt caching (for static instructions and taxonomy) and just-in-time feature materialization (updating affinities only upon new user actions) reduced overall computational costs by approximately 80%.

3. Experimental Evaluation

3.1 Offline Evaluation

The proposed model was evaluated against a baseline MTL model trained exclusively on historical engagement and item attributes using a 15-day holdout dataset.

• Metrics: Performance was measured using Area Under the Receiver Operating Characteristic Curve (AUC-ROC) and Mean Reciprocal Rank (MRR).
• Cohort Analysis:
  • Overall Population: The proposed model achieved a 4.4% relative lift in AUC-ROC and a 4.8% lift in MRR.
  • Cold-Start Consumers: This cohort saw a 4.0% lift in AUC-ROC and a 1.1% lift in MRR, validating the hypothesis that historical taste preferences can be effectively transferred across verticals.
  • Power Consumers: This group benefited most from the search query signal, achieving a 5.2% lift in AUC-ROC and a 2.2% lift in MRR.

3.2 Online Shadow Deployment

To validate offline findings, the system was deployed in a shadow mode against live traffic.

• Results: The LLM-augmented model demonstrated a 4.3% improvement in AUC-ROC and a 3.2% increase in MRR over the baseline for the general population, confirming real-world efficacy.

3.3 Feature Quality Assessment

Qualitative and quantitative evaluations confirmed the relevance of the generated features:

• Human Evaluation: Features derived from search queries were rated as "Highly Personalized" in 70.7% of cases, compared to 53.0% for restaurant orders.
• LLM-as-a-Judge: Using GPT-4o to score relevance, search queries again outperformed order history (61.6% highly personalized vs. 56.6%), aligning with the explicit nature of search intent.

4. Key Contributions and Significance

The paper claims to offer a practical path toward building more holistic and adaptive recommender systems by addressing the behavioral data gap in multi-vertical marketplaces.

• Novel Framework: The introduction of a hierarchical RAG pipeline to derive multi-level taxonomic features from unstructured user data.
• Semantic Bridge: Demonstrating that LLMs can effectively act as a semantic bridge, transferring knowledge from data-rich verticals (restaurants) to data-sparse ones (grocery/retail) to solve the cold-start problem.
• Production Viability: The work proves that LLM-generated features can be integrated into production MTL ranking models with significant performance gains while managing computational costs through strategic model selection and caching.
• Behavioral Insight: The results validate that latent user affinities inferred from one vertical can significantly improve personalization in another, particularly for users with no direct history in the target domain.

5. Future Work

The authors outline three specific directions for future research:

1. Advanced Prompting: Investigating paradigms such as Chain-of-Thought and self-correction, and exploring domain-specific fine-tuning of smaller, more efficient LLMs.
2. Funnel Integration: Integrating generative features earlier in the recommendation funnel, specifically within the candidate retrieval stage, to create a more synergistic end-to-end system.
3. Temporal Dynamics: Exploring the temporal dynamics of affinities to build more adaptive, session-aware recommendation models.