SOLARIS: Speculative Offloading of Latent-bAsed Representation for Inference Scaling

SOLARIS is a novel framework that enables real-time inference for complex foundation models in recommendation systems by proactively precomputing and asynchronously offloading user-item embeddings, thereby achieving significant revenue gains at Meta's scale without compromising serving latency.

The Gist

Imagine you are running a massive, high-stakes restaurant that serves millions of customers every single day. Your goal is to recommend the perfect dish to every guest instantly.

In the past, you had two choices:

1. The Master Chef (Foundation Model): A genius chef who knows every flavor combination in the world and can create the perfect recommendation. But this chef is slow, expensive, and takes too long to think. If you wait for them to decide, your customers will get angry and leave.
2. The Line Cook (Vertical Model): A fast, efficient cook who can make decisions in a split second. But they only know a few basic recipes and often miss the nuances that make a dish truly special.

Traditionally, restaurants tried to teach the Line Cook by having the Master Chef write down a "cheat sheet" of answers before the rush started. But this cheat sheet was often incomplete, outdated by the time the customers arrived, or only covered a small fraction of the menu.

Enter SOLARIS.

SOLARIS is a new system Meta AI built to solve this exact problem. Think of it as a super-smart, futuristic kitchen assistant that uses a technique called "Speculative Offloading." Here is how it works in plain English:

1. The "Crystal Ball" Prediction (Speculative Precomputation)

Instead of waiting for a customer to order and then asking the Master Chef to think, SOLARIS uses a "crystal ball." It looks at what customers are likely to order in the next few minutes based on trends and history.

• The Analogy: Imagine the kitchen assistant predicting that 50 people will likely order "Spicy Tacos" in the next hour. Instead of waiting, the assistant quietly asks the Master Chef to prepare the perfect Spicy Taco recommendation right now, while the kitchen is still calm.
• The Magic: This happens in the background, asynchronously. The Master Chef does the hard work when there is no pressure. The result is stored in a "smart pantry" (a cache) ready for immediate use.

2. The "Smart Pantry" (Asynchronous Offloading)

When a real customer actually walks up to the counter and asks for a recommendation:

• The Hit: If the system predicted correctly and the "Spicy Taco" recommendation is already in the smart pantry, the Line Cook grabs it instantly. The customer gets the Master Chef's genius advice with zero wait time.
• The Miss: If the customer orders something the crystal ball didn't predict (e.g., "Vegan Sushi"), the pantry is empty. The system doesn't panic. It just uses a placeholder (a blank note) for now and quietly asks the Master Chef to prepare that specific recommendation for next time.

3. The "Safety Net" (Hierarchical Feature Enrichment)

What if the pantry is empty and the system has no idea what the customer wants? SOLARIS has two backup plans to ensure the Line Cook never has to guess blindly:

• Plan A: The "Regulars" List (Aggregated Embeddings): Even if we don't know exactly what this customer wants for this specific dish, we know what they liked in the last 24 hours. SOLARIS averages out their recent preferences to give a "good enough" recommendation.
• Plan B: The "Lookalike" Strategy (Similarity-Based): If we have no data on this customer, SOLARIS looks for a "twin"—someone with very similar tastes. It says, "We don't know what you want, but your twin loves this dish, so you probably will too."

Why This Matters

Before SOLARIS, Meta had to choose between speed and quality. They either used the slow Master Chef (too expensive) or the fast but dumb Line Cook (poor recommendations).

With SOLARIS:

• Speed: The Line Cook still works at lightning speed because the hard thinking was done in the background.
• Quality: The Line Cook now has access to the Master Chef's genius knowledge 90% of the time (up from 50% previously).
• Result: Meta's advertising system got smarter without getting slower. This led to a 0.67% increase in revenue, which translates to roughly $100 million in extra value.

The Bottom Line

SOLARIS is like having a genius consultant who does their homework before the meeting starts. When the meeting begins, the consultant is ready to give you the perfect answer instantly. If they missed a question, they have a backup plan using your past notes or your friends' opinions. It's a way to get the best of both worlds: the brainpower of a giant AI and the speed of a real-time app.

Technical Summary

1. Problem Statement

Modern recommendation systems (RecSys) are increasingly relying on Foundation Models (FMs) due to their superior ability to learn complex user-item interactions from massive, heterogeneous datasets. However, deploying these FMs in real-time production environments faces two critical bottlenecks:

• Computational Latency: The high complexity of FMs makes real-time inference impractical for serving billions of requests with strict latency requirements.
• Inefficiency of Current Solutions: The industry standard, Knowledge Distillation (KD), forces a trade-off between quality and efficiency. Traditional KD relies on soft-label transfer during training, which suffers from:
  • Low Transfer Ratio: Typically achieves only a 20–25% knowledge transfer ratio.
  • Limited Generalizability: Soft labels are often tightly coupled to specific tasks.
  • Inference-Time Gap: Knowledge transfer occurs only during training, missing the opportunity to leverage FM representations during actual inference (serving).

2. Methodology: SOLARIS Framework

SOLARIS (Speculative Offloading of Latent-bAsed Representation for Inference Scaling) is a novel framework inspired by speculative decoding in Large Language Models (LLMs). It decouples the expensive FM inference from the latency-critical serving path by proactively precomputing representations.

Core Architecture

The system operates through three main mechanisms:

A. Direct Embedding-Based Transfer
Instead of transferring soft labels, SOLARIS extracts user-item interaction embeddings directly from the intermediate layers of the Foundation Model.

• Source: Embeddings are extracted from the last shared layer of the FM architecture, where holistic knowledge is concentrated.
• Compression: These embeddings are compressed via an autoencoder into fixed-dimensional vectors.
• Usage: These vectors serve as additional input features for downstream Vertical Models (VMs) (smaller, specialized models) during both training and inference.

B. Speculative Asynchronous Precomputation
To avoid the latency of running the FM in real-time, SOLARIS employs a background service to "speculate" and precompute embeddings for likely future requests.

• Verifier Model Mechanism: A lightweight verifier (the current VM's prediction score) filters candidate user-item pairs. Only pairs with high predicted relevance (top 20%) are selected for precomputation.
• Background Processing: The FM generates embeddings for these selected pairs asynchronously and stores them in a distributed cache keyed by <user, item>.
• TTL-Based Invalidation: To ensure freshness, the system enforces a Time-To-Live (TTL) on cached embeddings (typically a few hours), leveraging the observation that user preferences remain relatively stable over short periods.
• Fallback: If a pair is missing or expired in the cache, the VM uses zero tensors as placeholders while the pair is queued for the next refresh cycle.

C. Hierarchical Feature Enrichment (Coverage Boosting)
Since precomputing all possible user-item pairs is impossible (covering only ~50% of traffic initially), SOLARIS introduces two strategies to fill coverage gaps without linear cost increases:

1. Aggregated User-Only Embedding: For missing pairs, the system averages available user-item embeddings for that specific user over the past 24 hours. This leverages short-term user preference consistency, boosting coverage to ~85–90%.
2. Similarity-Based Embedding: For users/items without direct embeddings, the system uses a precomputed user clustering table (generated via KNN on FM embeddings). It retrieves embeddings from the most similar "neighbor" users. This extends effective coverage by an additional 30%.

3. Key Contributions

1. Inference-Time Knowledge Distillation: SOLARIS breaks the training-only limitation of traditional KD by enabling continuous, real-time knowledge transfer from FMs to VMs via precomputed embeddings.
2. High Transfer Ratio: By transferring rich latent representations rather than soft labels, SOLARIS achieves a 42–44% transfer ratio, representing a 2x improvement over previous distillation methods.
3. Speculative Offloading: The framework successfully decouples heavy FM computation from the critical serving path, allowing the use of models previously deemed too expensive for online use.
4. Scalable Coverage Strategy: The hierarchical enrichment approach (Aggregation + Similarity) solves the "long-tail" coverage problem, ensuring most users receive FM-enhanced features without prohibitive infrastructure costs.

4. Experimental Results

Deployed across Meta's advertising system serving billions of daily requests, SOLARIS demonstrated significant improvements:

• Model Performance:
  • Achieved relative Binary Cross-Entropy (BCE) loss reductions across 10+ production models.
  • CTR Tasks: 0.05%–0.09% loss reduction across Facebook Feed, Reels, and Instagram surfaces.
  • CVR Tasks: 0.05%–0.10% loss reduction for on-platform and offsite conversions.
• Business Impact:
  • Revenue Gain: Resulted in a 0.67% global ads revenue increase (estimated at the order of $100M).
  • Coverage: Increased feature coverage by 30%, directly correlating with performance gains.
• Efficiency:
  • Achieved these gains with zero additional latency overhead for the real-time serving path, as the heavy lifting was offloaded to the background.

5. Significance

SOLARIS represents a paradigm shift in scaling recommendation systems. It proves that Foundation Models can be effectively integrated into latency-sensitive, real-time production environments without sacrificing speed. By moving from "soft-label distillation" to "latent representation offloading," the framework unlocks the full modeling capacity of FMs for downstream tasks.

This approach is particularly significant for large-scale platforms (like Meta) where the gap between offline model performance and online serving constraints has historically forced a compromise on model complexity. SOLARIS bridges this gap, enabling richer personalization and higher revenue generation at scale.