Technical Summary: Fine-Tuned LLM as a Complementary Predictor Improving Ads System

Problem Statement

While Large Language Models (LLMs) have advanced rapidly, their direct integration into production-scale Recommendation Systems (RecSys), particularly for advertising, remains challenging. Industrial ad systems rely heavily on sparse, ID-centric features (user IDs, advertiser IDs) that do not naturally align with token-based semantic representations. Furthermore, existing LLM applications in RecSys typically fall into three categories: generative retrieval, late-stage re-ranking, or auxiliary signal enrichment. However, these approaches often struggle with the specific constraints of ad systems, including strict latency budgets, the dominance of sparse features, and the need to model complex targeting constraints and conversion histories. The core problem addressed is how to leverage the world knowledge and predictive capacity of LLMs to improve ad retrieval and ranking without replacing the existing, highly optimized industrial stacks or incurring prohibitive inference costs.

Methodology

The authors propose a complementary paradigm where a fine-tuned open-source LLM acts not as a direct ranker, but as an ads-specific ancillary predictor. The system forecasts likely advertisers and user interests conditioned on structured user profiles and conversion histories. These predictions serve as informative priors for downstream retrieval and ranking stages.

System Architecture

The pipeline consists of four stages:

1. User Selection: Inference is restricted to high-value segments (active U.S. users with recent conversions) to manage latency and cost.
2. Feature Compilation: Heterogeneous data (onsite/offsite searches, attributed/matched conversions, URLs, demographics) is compiled into a compact textual representation.
3. Prompt-Based Advertiser Prediction: The LLM is post-trained to predict a ranked list of advertisers and user interests.
4. Downstream Consumption: Predicted advertisers are used as targeting filters in the retrieval stage (Candidate Generation) and as features in conversion models (Ranking).

Post-Training Strategies

The paper explores two primary post-training approaches:

• Supervised Fine-Tuning (SFT): The model is trained to predict a single "next advertiser" based on a structured prompt containing user profiles, past conversions, and a preset advertiser pool.
• GRPO (Group Relative Policy Optimization): Following SFT, the model is optimized using reinforcement learning. The prompt asks for a ranked list of 20 advertisers and 5 user interests. The reward function ($R_{total}$) balances matching the ground-truth advertiser (with bonuses for top-rank positions) against length penalties for deviating from the required output format (20 advertisers, 5 interests). This encourages both accuracy and strict adherence to the structured XML output format required for production integration.

Semantic ID (SID) Enhancement

To bridge the gap between text semantics and ID-based ad systems, the authors incorporate Semantic IDs (SIDs) derived from multimodal embeddings (PinCLIP) via RQ-VAE. A multi-phase training approach is used to prevent catastrophic forgetting:

1. Phase 1: Align SID tokens with the text space by updating only SID embeddings while freezing the rest of the model.
2. Phase 2: Full-parameter pre-training to incorporate recommendation knowledge while mixing in general-domain data.
3. Phase 3: Task-specific fine-tuning (SFT/GRPO) where prompts include both text features and recent SID sequences.

Downstream Integration

• Retrieval: The LLM-predicted advertisers act as a complementary candidate generator (CG), filtering the retrieval space to prioritize advertisers with high conversion intent. This is particularly effective for users with weak existing signal coverage.
• Ranking: The predicted advertisers and interests are featurized and fed into downstream conversion models (ctcvr, vtcvr).

Key Results

Offline Performance

• Prediction Quality: On the V1 dataset (aligned with production traffic), the combination of SFT (1-advertiser prompt) followed by GRPO significantly outperformed zero-shot prompting. The inclusion of Semantic IDs provided substantial gains, improving Recall@1, Recall@5, and Recall@20 by over 10% compared to text-only models.
• Feature Ablation: The most critical input feature was "active advertisers with past conversions." Offsite URLs and offsite searches were the next most influential features. Interestingly, explicit reasoning chains were found to be less useful than direct SFT-style training for this specific intent-aggregation task.
• Conversion Models: Adding LLM-derived advertiser features to conversion models resulted in measurable offline improvements, particularly in PR-AUC (Precision-Recall AUC), which is crucial for sparse conversion settings.

Online Business Impact

• Retrieval: A/B testing on U.S. Shopping Ads showed that the LLM-based candidate generator increased Return on Ad Spend (RoAS) by 4.94% for the general treatment slice and 6.69% for the opt-in slice.
• Mechanism: The gains were driven by the system surfacing traffic from advertisers with high conversion intent that were previously under-covered by standard collaborative filtering channels.
• Operational Constraints: The study highlighted that careful tuning of the candidate generator's quota and the training objective (using clicks with duration-based weights rather than raw impressions or sparse conversions) was essential to balance funnel survival and engagement.

Significance and Claims

The paper claims to demonstrate a practical path for LLM-enhanced recommendation at scale by redefining the role of LLMs in industrial ad systems. Rather than attempting to replace the existing multi-stage retrieval and ranking stacks or using LLMs solely as expensive re-rankers, the authors show that fine-tuned LLMs can serve as complementary predictors.

Key contributions include:

1. Paradigm Shift: Successfully formulating an advertiser-prediction task that leverages LLM world knowledge to bridge the gap between text semantics and sparse ID features in ads.
2. Practical Recipe: Providing a validated workflow for data compilation, prompt design (including SID integration), and fine-tuning (SFT + GRPO) that yields stable, low-latency signals usable across the entire ad pipeline.
3. End-to-End Validation: Demonstrating that these LLM-derived signals can unlock measurable business value (RoAS improvements) by improving both retrieval coverage and ranking precision without disrupting the core infrastructure.

The authors remain modest, noting that while the multimodal SID extension showed strong offline results, it has not yet been fully validated in online production environments and is therefore excluded from the primary empirical claims. The work emphasizes that the success of this approach relies on the specific design of the ancillary prediction task and its tight integration with existing industrial constraints.