🤖 AI

SA-RSQ: A Versatile Sparse Representation Framework for Multi-modal Recommender Systems

The paper proposes SA-RSQ, a versatile framework for multi-modal recommender systems that utilizes sparse activation-based residual soft quantization to store compact (Index, Probability) tuples, effectively balancing storage efficiency and reconstruction quality while achieving significant improvements in CTR and CPM in industrial applications.

Original authors: Xiang Wang, Shigang Quan, Tingzhen Chang, Kang Yang, Sitong Chen, Yabo Fan, Xingxing Wang, Zhaodian He

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiang Wang, Shigang Quan, Tingzhen Chang, Kang Yang, Sitong Chen, Yabo Fan, Xingxing Wang, Zhaodian He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast digital marketplaces of today, recommender systems act as the invisible librarians of our lives, suggesting the next video to watch, the song to hear, or the meal to order. To do this well, these systems rely on a deep understanding of the items they offer. In recent years, engineers have begun using powerful artificial intelligence models to describe these items with incredibly detailed, high-dimensional maps. Imagine a single description of a product not as a simple label, but as a complex, multi-layered portrait containing thousands of distinct details about its appearance, meaning, and context. While these rich descriptions help the system understand subtle differences between similar items, they come with a heavy price. Storing and processing these massive portraits for billions of items requires so much computer memory and energy that it slows down the entire system, making it too slow and expensive for real-world use.

To solve this, engineers have traditionally tried to shrink these detailed portraits into tiny, discrete codes, much like compressing a high-resolution photograph into a single, tiny icon. However, this extreme compression often blurs the image, causing the system to lose the fine details that distinguish one item from another. It is a difficult trade-off: keep the rich detail and slow down the system, or shrink the data and lose the precision needed to make good recommendations. Researchers at Tianjin University and Meituan have proposed a new approach that attempts to find a middle ground, allowing the system to keep the richness of the detailed descriptions while storing them in a space-efficient way that does not sacrifice accuracy.

The researchers developed a method called Sparse Activation-based Residual Soft Quantization, or SA-RSQ. Instead of forcing every item into a single, rigid category or a tiny, fixed code, this new framework treats the item's description as a flexible combination of a few key building blocks. Think of it like describing a complex flavor not by picking a single word from a dictionary, but by selecting a small handful of ingredients and specifying exactly how much of each to use. The system looks at a high-dimensional description of an item and identifies the most relevant "ingredients" from a large library of possibilities. It then stores only the names of these selected ingredients and the precise proportions in which they are mixed.

This approach offers a significant advantage over previous methods. Older techniques often forced a choice between a single code or a dense block of numbers, leading to a loss of nuance or a spike in storage costs. The new method, however, decouples the amount of storage space from the complexity of the information. By storing only the most important parts of the description along with their weights, the system can reconstruct a highly accurate version of the original item portrait whenever needed. Crucially, this process is differentiable, meaning the system can learn and improve its choices directly from the feedback it receives during training, rather than relying on rough approximations that often lead to errors.

The team tested this framework on a massive, real-world dataset from a food-delivery advertising platform, involving hundreds of millions of items. They compared their method against several existing compression techniques under strict storage limits, ranging from 8 bytes to 48 bytes per item. The results showed that their approach consistently outperformed the others. Even when constrained to very small storage sizes, the new method maintained a higher level of accuracy in predicting what users would click on. When allowed slightly more space, such as 32 or 48 bytes, the performance improved further, achieving the highest scores among all tested methods. The system was able to preserve the fine-grained details of the items, preventing the "collisions" where different items get confused with one another, a common problem in older compression systems.

Beyond the offline tests, the researchers deployed the system in a live online experiment on the food-delivery platform. Over the course of one week, they ran a controlled test where the new method was shown to a portion of the actual user traffic. The results were tangible: the system using this new framework generated a 2.51 percent increase in the rate at which users clicked on ads, and a 3.66 percent increase in the revenue generated per thousand impressions. These gains were achieved without slowing down the system, proving that it is possible to compress complex data without losing the intelligence required to make smart recommendations.

The study also explored a potential future application where the system does not just predict a single next item, but predicts a probability distribution of what might come next, similar to how a language model predicts the next word in a sentence. While this was a preliminary investigation, early results suggested that this probabilistic approach could work well for generative recommendation tasks, opening a new path for how these systems might evolve. The researchers noted that while the results are promising, they are based on proprietary data and specific configurations, and further work is needed to confirm these findings across different domains.

Ultimately, this work demonstrates that the rigid trade-off between storage efficiency and data quality is not inevitable. By using a flexible, sparse representation that captures the essence of an item through a weighted combination of key features, it is possible to build recommender systems that are both fast and precise. The success of this method in a real-world industrial setting suggests that such techniques could become a standard tool for handling the massive amounts of data that power the digital world, ensuring that the systems guiding our choices remain as intelligent and nuanced as the information they process.

Technical Summary: SA-RSQ

1. Problem Statement

Industrial recommender systems are increasingly adopting high-dimensional semantic features (e.g., 2048+ dimensions) extracted by Multimodal Large Language Models (MLLMs) to improve cold-start handling and semantic understanding. However, deploying these dense vectors at scale incurs prohibitive storage and inference latency costs.

Existing solutions face a critical trade-off:

Extreme Compression (Hard Quantization): Methods like RQ-VAE compress features into short discrete Semantic IDs (e.g., 8 bytes). While storage-efficient, they suffer from boundary distortion (where semantically similar items are forced into identical or entirely different discrete IDs) and non-differentiability. The reliance on the Straight-Through Estimator (STE) for gradient approximation introduces bias, destabilizes training, and creates a fundamental misalignment between the reconstruction objective of the quantizer and the downstream recommendation objectives (CTR/CVR).
Soft Quantization: Approaches like SoftVQ-VAE avoid STE by using dense soft assignments. However, under strict storage budgets (e.g., 32 bytes), projecting high-dimensional vectors into such narrow bottlenecks often leads to severe semantic collapse. Conversely, increasing dimensions to preserve quality negates the storage benefits.

There is currently no principled mechanism to smoothly trade off between extreme compression and high-dimensional representation quality without incurring irreversible information loss or storage bloat.

2. Methodology: SA-RSQ

The authors propose Sparse Activation-based Residual Soft Quantization (SA-RSQ), a framework designed to decouple storage footprint from representation dimensionality while maintaining differentiability.

Core Architecture

SA-RSQ utilizes an Encoder-Decoder architecture with a cascaded residual structure similar to RQ-VAE but replaces the non-differentiable arg min hard assignment with a differentiable sparse soft routing mechanism.

Iterative Residual Formulation:
- Input embeddings are mapped to a latent space.
- The quantization process cascades through $L$ stages. At each stage $l$ , a residual vector $r_{l-1}$ is approximated by a weighted combination of codebook entries.
- The final reconstruction $\hat{x}$ is the sum of these approximations decoded by the decoder.
Differentiable Sparse Routing (Top- $K$ with Masked Softmax):
- Instead of applying Softmax over all $V$ code words (which is storage-prohibitive), SA-RSQ computes similarity logits and selects the Top- $K$ most relevant code words.
- A masking operation sets logits for non-selected words to $-\infty$ , followed by a Softmax to generate a sparse probability distribution $p^{(l)}$ .
- The quantized representation is the convex combination of the active code words weighted by these probabilities: $\hat{r}_{l-1} = \sum_{i \in \mathcal{I}_K} p^{(l)}_i c^{(l)}_i$ .
- Storage Format: Only the indices of the Top- $K$ support and their corresponding non-zero probabilities are stored as compact (Index, Prob) tuples. This decouples per-item storage from the codebook dimensionality.
Curriculum Sparsity Annealing:
- To prevent premature pruning and ensure global semantic exploration, the framework employs a Cosine Sparsity Annealing schedule. Training begins with fully dense activation ( $K=V$ ) and gradually decays to the target sparsity $K_{tgt}$ , balancing exploration and exploitation.

Optimization Objectives

The framework is trained using a joint objective function comprising four components:

Global & Layer-wise Reconstruction: Minimizes MSE between original and reconstructed embeddings, plus a layer-wise residual loss to stabilize cascaded training.
Geometric Regularization (Orthogonality): Enforces orthogonality among codebook vectors within each stage to maximize the spanning capacity of the linear combinations and eliminate spatial redundancy.
Mutual Information Regularization (MIR): Prevents codebook collapse by maximizing the mutual information between inputs and soft assignments. This involves minimizing sample entropy (encouraging sharp distributions per item) and maximizing batch entropy (ensuring uniform codebook usage).
Total Loss: $\mathcal{L}_{total} = \mathcal{L}_{recon} + \lambda_1 \mathcal{L}_{residual} + \lambda_2 \mathcal{L}_{ortho} + \lambda_3 \mathcal{L}_{MIR}$ .

Deployment Paradigms

Two-Stage Paradigm: Offline, sparse routing tuples are generated and frozen. Online, the model performs lightweight lookups and weighted summations over a trainable codebook. This allows CTR models to leverage high-fidelity semantics with negligible memory overhead.
End-to-End Alignment: Unlike hard IDs, the sparse probabilities serve as a differentiable medium. For a fixed Top- $K$ support, downstream gradients can backpropagate through the weighted sum to update the encoder and codebook without STE, aligning representation learning with downstream tasks.

3. Key Contributions

Differentiable Sparse Soft Quantization: SA-RSQ replaces heuristic STE-based routing with differentiable Top- $K$ sparse routing, enabling flexible storage-performance trade-offs (8–48 bytes per item) without irreversible distortion.
Superior Compression-Performance Trade-offs: The framework achieves favorable balances between storage efficiency and recommendation performance. It introduces a "Next-Distribution Prediction" formulation for generative recommendation, leveraging probabilistic representations.
Industrial Validation: The method was deployed in a Food Delivery Advertising Platform. Online A/B tests demonstrated relative improvements of +2.51% in CTR and +3.66% in CPM over the production baseline.

4. Experimental Results

Experiments were conducted on a proprietary food-delivery advertising dataset with hundreds of millions of items and 2048D MLLM embeddings.

Discriminative Task (CTR Prediction):
- 8 Bytes: SA-RSQ (Index only, $L=4, K=1$ ) achieved the highest AUC (64.616%) among 8-byte baselines, outperforming RQ-VAE and R-Kmeans.
- 32 & 48 Bytes: When storing full (Index, Prob) tuples, SA-RSQ consistently outperformed dense embedding baselines and other quantization methods. At 48 bytes, it achieved the highest AUC (64.913%) and gAUC (63.013%).
- Reconstruction Quality: SA-RSQ significantly reduced Reconstruction Loss (RL) to 0.1553 (at 48 bytes) compared to >0.3 for traditional methods, indicating better preservation of fine-grained semantics.
Generative Recommendation (Preliminary Study):
- A study comparing "Next Token Prediction" (NTP) with the proposed "Next Distribution Prediction" (NDP) showed that NDP using SA-RSQ's sparse probabilities yielded further gains in Recall@K and NDCG@K over discrete ID baselines.
Ablation Studies:
- Removing Mutual Information Regularization caused catastrophic codebook collapse (usage dropped to 24.5%).
- Removing Curriculum Learning led to poor exploration (Usage 15.9%).
- Removing Proxy Co-training significantly degraded Semantic Cohesion, highlighting the importance of aligning offline compression with collaborative signals.

5. Significance and Claims

The paper claims that SA-RSQ effectively bridges the gap between extreme hard compression and memory-intensive dense embeddings by establishing a measurable and differentiable semantic space.

Decoupling: It successfully decouples storage constraints from representation dimensionality, allowing systems to store compact tuples while reconstructing high-dimensional features on the fly.
Differentiability: By avoiding the Straight-Through Estimator, it enables true end-to-end alignment between representation learning and downstream objectives.
Practicality: The framework has been validated in a real-world industrial setting with measurable business metric improvements.

Limitations and Modesty:
The authors explicitly note limitations in their claims:

The results are based on a proprietary dataset and do not claim transferability to public benchmarks without further evaluation.
The "Next-Distribution Prediction" paradigm is presented as a preliminary study, not a proven general paradigm.
The paper acknowledges a lack of public data, repeated-run uncertainty, and detailed latency statistics (p99), which limits the strength of broader claims regarding system-level performance.
The reported online A/B test results are peak values from a specific configuration and should be interpreted as deployment evidence rather than a rigorous significance analysis due to withheld traffic counts and confidence intervals.