🤖 machine learning

PaletteID: Prototype-Composed Semantic Identifiers for Multimodal CTR Prediction

This paper proposes PaletteID, a prototype-based semantic identifier that leverages a diverse set of representative prototype items to bridge pretrained multimodal embeddings and recommendation models, thereby overcoming the limitations of existing discrete identifier methods to improve CTR prediction accuracy, particularly for long-tail items.

Original authors: Huanyu Liu, Baining Chen, Hui Liu, Zengyang Li, Ziyi Huang

Published 2026-08-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Huanyu Liu, Baining Chen, Hui Liu, Zengyang Li, Ziyi Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a massive, endless library where every book has a unique barcode. In the world of online shopping and video streaming, computers act like librarians trying to guess which book you'll love next. To do this, they look at your past choices and the "barcodes" (identifiers) of the items you've clicked on. But here's the problem: these barcodes are often just random numbers that mean nothing to a human. If a new book arrives that no one has ever seen before, the computer gets confused because it has no history for that specific barcode.

To fix this, scientists have tried to give items "meaningful" barcodes based on what they actually look like or say. They take a picture or a description of an item and turn it into a list of short, discrete codes, kind of like turning a complex painting into a simple set of primary colors. This helps the computer understand that a red apple and a red ball are somewhat related because they share the "red" code. However, the old way of doing this is a bit rigid. It's like trying to describe a sunset by forcing it to be only "Red" or only "Orange," with no in-between. If the sunset is a tiny bit more pink, the old system might suddenly switch the whole description to "Pink," losing all the nuance. This paper tackles that rigidity, suggesting a smarter way to mix these semantic "colors" so the computer can understand the subtle shades of what we like.

The Palette Problem: Why Old Barcodes Are Boring

The researchers behind this study, led by Huanyu Liu and colleagues, noticed that the current method for creating these "meaningful barcodes" (called Semantic IDs) has two big flaws. First, it's too black-and-white. It forces an item into a single, hard bucket, throwing away the fine details that make an item unique. Second, it's like building a tower of blocks where every new block depends entirely on the one below it. If you change the bottom block, the whole tower wobbles, making the system unstable and hard to scale up for millions of items.

The team asked: What if we didn't force an item into just one bucket? What if, instead, we described every item as a unique mix of several "master" items?

Enter PaletteID: The Art of Mixing Colors

The authors propose a new system called PaletteID (or PID for short). They draw inspiration from how artists use a limited set of paint colors on a palette to create infinite, rich paintings. Instead of picking just one "code" for an item, PID picks a small group of representative "prototype" items and blends them together.

Here is how the magic happens, step by step:

Creating the Master Palette: First, the system scans the entire library of items and picks a special, compact set of "prototype" items. These aren't random; they are chosen using a smart mathematical trick called SQ-DPP. This method ensures the palette has a good mix of different types of items (diversity) while making sure the chosen items are actually popular and representative of their groups (quality). Think of this as an art teacher picking the perfect set of 500 master paintings to represent every style in the world, from landscapes to portraits, without picking 500 paintings of the same sunset.
Mixing the Colors: When the system needs to describe a specific item (like a new video about "handmade jewelry"), it doesn't just pick one match. It looks at the master palette and finds the top 12 to 15 most similar prototypes. Maybe one prototype is a "handmade accessory," another is a "small business," and a third is a "promotional video."
The Secret Sauce (Soft Weights): This is where PID beats the old methods. Instead of just saying "This item is 100% Prototype A," PID calculates exactly how much of each prototype to use. It uses the actual similarity between the item and the prototypes to assign a "weight." If the item is very similar to the "handmade" prototype but only a little bit like the "business" one, the system gives the "handmade" prototype a heavy weight and the "business" one a light touch. It's like mixing 70% red paint with 30% yellow paint to get a perfect orange, rather than just shouting "Red!" or "Yellow!"

What They Found: Smarter, Stronger, and More Stable

The researchers tested this new system on two huge real-world datasets: one from Taobao (a massive Chinese shopping site with 1.0 million items and 7.2 million users) and another from Kuaishou (a video app with 10.7 thousand items). They compared their PaletteID against the standard "hard code" methods.

The results were promising. The paper suggests that PaletteID consistently improves the accuracy of predicting whether a user will click on an item. But the real win was for the "long-tail" items—the obscure, rare, or new things that don't have many clicks yet. Because PaletteID can blend multiple semantic ideas, it helps the computer understand these rare items much better than the old rigid systems could. For example, on the Taobao dataset, the new method improved the prediction score for the rarest items by a noticeable margin compared to the old methods.

The authors also found that PaletteID is much more robust. If you slightly tweak the description of an item (like changing a word in the title), the old system might suddenly jump to a completely different code, confusing the model. PaletteID, however, changes its "mix" smoothly. If the item becomes slightly more "business-like," the weight of the business prototype just increases a little bit, rather than the whole identity flipping upside down.

Why It Matters

This approach offers a more flexible and interpretable way to teach computers about the world. Instead of hiding an item's meaning behind a cryptic code like "Token #492," PaletteID lets us see exactly which real-world examples the computer is using to understand the item. It's a shift from forcing items into rigid boxes to letting them be a unique, blended composition of the things they resemble.

The paper concludes that while this system requires some offline work to build the palette and calculate the mixes, it is very fast to use in real-time. It suggests that by treating items as a "palette" of prototypes rather than a single code, we can build recommendation systems that are not only more accurate but also better at understanding the subtle, complex reasons why we click on what we click.

Technical Summary: PaletteID (Prototype-Composed Semantic Identifiers for Multimodal CTR Prediction)

1. Problem Statement

Multimodal information (e.g., images, text, video) holds significant potential for improving Click-Through Rate (CTR) prediction, particularly for cold-start and long-tail items where sparse interaction data exists. While recent approaches discretize pretrained multimodal embeddings into Semantic Identifiers (SIDs) to bridge content and behavior, existing methods face two critical limitations:

Loss of Continuous Semantic Signals: Standard hard codebook assignment (e.g., in VQ-VAE or RQ-KMeans) forces items into discrete bins, discarding fine-grained continuous signals and making it difficult to distinguish varying degrees of semantic relatedness between items.
Scalability and Ambiguity in Residual Codes: Hierarchical residual SIDs suffer from prefix dependency. The same code index at deeper levels may represent different semantics depending on the prefix, leading to inconsistent collaborative signals. Furthermore, creating distinct embeddings for every code path causes exponential parameter growth, while sharing embeddings across prefixes introduces semantic ambiguity. Additionally, hard identifiers are unstable; minor perturbations in the embedding space can cause abrupt identifier changes, especially at deeper quantization levels.

2. Methodology: PaletteID (PID)

The authors propose PaletteID (PID), a prototype-based semantic identifier inspired by color composition. Instead of assigning a single hard code, PID represents each item as a soft composition of multiple representative semantic prototypes.

2.1 Prototype Selection via SQ-DPP

The core of PID is a "prototype palette"—a compact set of representative items selected from the multimodal embedding space.

Semantic Quality-Aware Determinantal Point Process (SQ-DPP): Unlike standard clustering or reconstruction methods, PID selects real items as semantic anchors.
Cosine-RBF Kernel: To avoid rank bottlenecks in high-dimensional spaces, the authors construct a kernel using Cosine similarity mapped through an RBF function ( $K_{ij} = \exp(-\gamma(1 - \cos(\hat{x}_i, \hat{x}_j)))$ ).
Quality Scoring: A content-density score ( $\rho_i$ ) is calculated using Approximate Nearest Neighbor (ANN) search. This score guides the selection toward semantically dense and representative regions, preventing the selection of isolated items.
Optimization: The final kernel $L_{ij} = q_i K_{ij} q_j$ combines diversity (via the determinant of $K$ ) and local representativeness (via quality scores $q$ ). A greedy MAP inference algorithm selects the optimal subset of $M$ prototypes offline.

2.2 PID Embedding Generation

For a target item $i$ :

Retrieval: The system retrieves the top- $K$ most similar prototypes from the palette based on multimodal similarity.
Aggregation: The corresponding trainable embeddings of these prototypes are aggregated.
Soft Weighting: Unlike hard assignment, PID uses a lightweight affine transformation on the similarity scores to generate soft weights ( $\alpha_i$ ). A monotonic regularization term ensures that higher-ranked prototypes receive equal or greater confidence.
Composition: The final PID embedding is a weighted sum: $e^{PID}_i = \alpha_i^T E_i$ , preserving fine-grained continuous proximity.

2.3 Downstream Integration

The PID embeddings serve as additional item-side features in standard CTR backbones (e.g., DeepFM, DCNv2, DIN). The training objective combines the standard binary cross-entropy loss for CTR with the monotonic regularization loss.

3. Key Contributions

Prototype-Based Identifier: The paper proposes PID, which moves away from discrete codebook assignment to a soft, prototype-composition approach. This preserves multimodal similarity while offering higher compositional capacity.
SQ-DPP for Palette Construction: A novel method for selecting real-item prototypes that jointly optimizes for global semantic diversity and local content density, avoiding the pitfalls of pure clustering or random sampling.
Empirical Validation: Extensive experiments on two public datasets (TAOBAO-MM and KuaiRec) demonstrate that PID outperforms existing residual SID methods in CTR prediction, long-tail generalization, robustness, and interpretability.

4. Experimental Results

The authors evaluated PID against baselines (VQ-VAE, RQ-VAE, RQ-KMeans) across different CTR backbones (DCNv2, RankMixer).

CTR Prediction: PID consistently achieved the highest AUC and GAUC scores. For instance, on TAOBAO-MM with DCNv2+, PID achieved an AUC of 0.6314, outperforming the best SID baseline (RQ-VAE at 0.6297).
Long-Tail Performance: The gains were most pronounced for long-tail items. PID improved AUC by +0.0128 for the Tail group (40% of items) compared to the baseline, significantly outperforming RQ-VAE (+0.0098). This suggests PID effectively transfers semantic knowledge to items with sparse interactions.
Robustness: Under Gaussian noise perturbations in input embeddings, PID maintained significantly higher assignment stability (Assignment Cosine) compared to residual SID methods. While SID consistency dropped sharply with noise, PID's weighted prototype mixture remained stable.
Scalability: The required palette size ( $M$ ) scales with semantic complexity rather than raw catalog size. On TAOBAO-MM (1M items), an optimal palette size of 650 was sufficient, suggesting PID does not require linear growth in parameters as the item catalog expands.
Interpretability: Case studies showed that PID prototypes correspond to real, interpretable content facets (e.g., "handcraft," "business," "promotion"), whereas SID codes are abstract indices.

5. Significance and Claims

The paper claims that PaletteID addresses the fundamental trade-off between the efficiency of discrete identifiers and the expressiveness of continuous embeddings. By using a soft composition of real-item prototypes, PID:

Bridges the Gap: It effectively connects pretrained multimodal semantics with behavior-driven CTR objectives without losing fine-grained information.
Enhances Generalization: It provides a robust mechanism for long-tail recommendation by allowing items to share knowledge through multiple complementary semantic anchors rather than a single rigid code.
Improves Stability: It mitigates the instability of hard quantization, where minor input changes lead to drastic identifier shifts.
Offers Interpretability: Unlike abstract codebook indices, PID's components are traceable to concrete prototype items, making the semantic representation inspectable.

The authors conclude that PID offers a scalable, robust, and interpretable alternative to existing SID methods, suitable for large-scale industrial recommendation systems where offline prototype selection and online lightweight aggregation can be efficiently deployed.