🤖 machine learning

MMRM: A Multiplex Multimodal Representation Model for Product Ranking in E-commerce Search

The paper proposes MMRM, a unified framework that aligns Multimodal Large Language Models with diverse collaborative signals to generate multiplex item and user representations, significantly improving e-commerce search ranking performance and efficiency in the JD search engine.

Original authors: Zhen-Lin Chen, Maosen Sheng, Peng Lin, Jianmin Chen, Zhuojian Xiao, Dongyue Wang, Xiwei Zhao

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhen-Lin Chen, Maosen Sheng, Peng Lin, Jianmin Chen, Zhuojian Xiao, Dongyue Wang, Xiwei Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, super-fast digital mall (like JD.com) where millions of people are hunting for the perfect pair of jeans. To help them find what they want, you need a smart assistant that doesn't just read the text on a product tag but also "sees" the picture, understands the style, and knows exactly what the shopper is looking for.

For a long time, these assistants had a tricky problem. They were like students who only studied for one specific test at a time. If they studied for "Search," they were great at matching a text query to a product. If they studied for "Cart," they were good at knowing what people put in their shopping carts. But trying to make one student study for all these different tests at once usually resulted in a confused mess. The old methods tried to force a single "super-student" (a general Multimodal Large Language Model) to learn from just one type of signal, or they treated the student's notes as a generic list of facts that didn't really help predict what a specific user wanted next.

The paper introduces a new system called MMRM (Multiplex Multimodal Representation Model) that solves this by acting like a master chef with a single, high-tech kitchen but four different, specialized recipe books.

The Big Idea: One Kitchen, Four Specialized Chefs
Instead of building four separate kitchens (models) for every task, MMRM uses one shared "backbone" (the kitchen itself) but gives it four special "hats" or tokens: [SEARCH], [CLICK], [CART], and [ORDER].

Think of it like a versatile actor. When they put on the [SEARCH] hat, they act like a detective matching a text clue to a product. When they switch to the [CART] hat, they act like a stylist who knows which items people actually buy together. The magic is that this actor can switch hats instantly without needing to leave the stage. In one single trip through the kitchen, the model learns from four different types of clues (signals) at the same time:

Search clicks: When someone types "jeans" and clicks a result.
Click sessions: When someone clicks from one item to another in a row.
Cart sessions: When someone adds items to their cart in a sequence.
Order sessions: When someone actually buys a sequence of items.

The authors found that these signals are very different. A search click is like a broad "I'm interested," while an order session is a "I really, really want this specific combo." By ignoring the differences and treating them all the same, old models missed out. MMRM, however, learns all four at once, creating four distinct "representations" (or mental maps) of the same product, each tuned for a different job.

The "Multiplex" User Strategy
Once the model has these four different maps of the products, it needs to figure out what you want. The paper argues that you can't just use one generic map for everyone. Instead, MMRM uses a "multiplex user representation" strategy.

Imagine you are browsing. The system looks at your history. If you are just searching, it uses the [SEARCH] map to find items similar to what you clicked. If you are building a cart, it switches to the [CART] map to see what usually goes together. It's like having a personal shopper who changes their strategy based on whether you are just window-shopping or ready to buy. They take your specific behavior sequence and match it against the right "hat" of the product, creating a highly personalized recommendation.

What They Ruled Out
The paper explicitly argues against two common approaches:

Using only one signal: They show that training a model on just search data (or just cart data) leaves it blind to the other important relationships. You can't just pick one and ignore the rest.
The "One-Size-Fits-All" embedding: They found that simply taking one generic product description and shoving it into a ranking model doesn't work well for multitask scenarios. It causes "embedding entanglement," where the model gets confused because it's trying to use the same note for a search query and a cart prediction simultaneously. MMRM rejects this by keeping the representations separate (disentangled) for each task.

The Proof: Real Numbers, Real Results
The authors didn't just guess; they tested this on a massive scale.

Training Data: They used a huge dataset of 0.3 billion triplets (groups of three related items) collected over six months for search, and even larger datasets for cart and order behaviors (up to 3 years of data for orders).
The Model: They trained a 4-billion parameter model (a very large AI) on 8 NVIDIA H800 GPUs.
The Results: In their tests, MMRM beat all the previous "single-task" models and the "multi-task" models that didn't use this special hat system.
Real World Impact: They didn't stop at computer simulations. They deployed this on the actual JD.com search engine. In a week-long test with millions of daily users, the new system improved:
- Click-Through Rate (CTR) by 0.42%
- Add-to-Cart Rate (ACR) by 0.37%
- Conversion Rate (CVR) by 0.35%

The authors note that while these percentages look small, in a platform with millions of users, even a 0.10% lift leads to massive revenue growth. Because of these proven gains, the system is now fully live and helping millions of people find their products.

In short, MMRM is a smarter, more flexible way to teach AI how to understand shopping. It stops trying to be a "jack of all trades" that masters none, and instead becomes a master of many trades by using a single, efficient brain with four specialized modes.

Technical Summary: MMRM for E-commerce Product Ranking

1. Problem Statement

E-commerce search ranking requires capturing fine-grained user preferences across multiple objectives (e.g., Click-Through Rate, Add-to-Cart Rate, Conversion Rate). While recent approaches fine-tune general Multimodal Large Language Models (MLLMs) to generate item representations, the authors identify two critical limitations in existing methods:

Reliance on Single Collaborative Signals: Current methods typically fine-tune MLLMs using only one type of signal (e.g., search-based query-to-item clicks or behavior-based item-to-item clicks). The authors argue that e-commerce signals are inherently heterogeneous. Search signals capture broad relevance, while sequential behaviors (cart, order) reflect stronger fine-grained similarities and complementary relationships. Ignoring this diversity prevents models from learning comprehensive representations necessary for multitask ranking.
Underutilization in Downstream Ranking: Existing ranking models often treat multimodal representations as regular item features, using a single embedding table for feature interaction or simple behavior grouping. This approach fails to capture "multiplex user intents," leading to information loss and embedding entanglement in multitask scenarios.

2. Methodology

The proposed solution, the Multiplex Multimodal Representation Model (MMRM), is a unified framework designed to align MLLMs with diverse collaborative signals and integrate them into multitask ranking systems.

2.1 Dataset Construction

The authors constructed four distinct triplet datasets to capture heterogeneous signals:

Q2I Dataset (Search): Built from query-item clicks ( $q2i\_click$ ), representing the most pervasive but coarse-grained signal. Hard negatives are sampled from the same parent category but different sub-categories.
I2I Datasets (Behavior): Three datasets ( $i2i\_click$ $i 2 i_c l i c k$ , $i2i\_cart$ $i 2 i_c a r t$ , $i2i\_order$ $i 2 i_or d er$ ) are constructed via graph-based approaches.
- Items are connected based on sequential user behaviors within specific time windows (e.g., 10 minutes for clicks, 30 minutes for carts).
- To mitigate hot-item dominance, anchor items are subsampled using a probability function similar to word2vec.
- Hard negatives are sampled from the same sub-category as the positive item to enhance contrastive learning.

2.2 MMRM Architecture

MMRM employs a shared MLLM backbone (initialized from Qwen3-VL-4B-Instruct) with specific architectural modifications to handle multitask learning:

Task-Specific Tokens: Four special tokens—[SEARCH], [CLICK], [CART], and [ORDER]—are appended to inputs to disentangle tasks.
Projection Layers: The hidden state of the specific task token is projected via a task-specific Multi-Layer Perceptron (MLP) to generate the final task-specific item representation ( $r^t_i$ ).
Training Objective: The model is trained on a uniform mix of the four datasets. For each task $t$ , the contrastive learning loss is calculated using positive pairs from that task, while items from other tasks serve as in-batch negatives. This allows the model to learn from diverse signals simultaneously without performance degradation.

2.3 Multitask Ranking Application

The authors introduce a multiplex user representation strategy for the downstream ranking model:

Soft Search: For a given query and target item, the system extracts a top- $K$ relevant behavior subsequence using the multimodal embedding table specific to the current task (e.g., using [CART] embeddings for ACR prediction).
User Representation: Task-specific user representations ( $U_t$ ) are derived using a Multi-Head Target Attention (MHTA) module over the filtered behavior sequence.
MMoE Integration: All task-specific user representations and other features are processed by a Multi-gate Mixture-of-Experts (MMoE) module to yield task-specific hidden states.
Prediction: These states are fed into task-specific towers (CTR, ACR, CVR) to generate final predictions.

3. Key Contributions

Unified Framework: MMRM effectively aligns MLLMs with four diverse collaborative signals (search, click, cart, order) using a shared backbone and task-specific tokens, enabling the efficient generation of multiplex item representations in a single inference pass.
Multiplex User Strategy: The paper proposes a novel strategy to derive task-specific user representations from multiplex item representations, addressing the issue of embedding entanglement in multitask ranking.
Scalability: The authors demonstrate that MMRM can be trained efficiently on large-scale data (0.3 billion triplets) using techniques like GradCache to handle large batch sizes on modern MLLMs.

4. Experimental Results

4.1 Representation Model Evaluation

Performance: MMRM consistently outperformed single-task baselines and a "Vanilla-Multi" baseline (which used shared embeddings with task prompts).
Efficiency: MMRM generates all four task embeddings in a single forward pass, whereas baselines required four separate passes.
Ablation: Experiments confirmed that larger batch sizes (achieved via GradCache) significantly boost performance, validating the importance of diverse in-batch negatives.

4.2 Ranking Model Evaluation

Offline: The proposed ranking model using multiplex filtering ( $SIM_{soft}(item_{MMRM[ALL]})$ ) achieved the best GAUC scores across CTR, ACR, and CVR tasks compared to baselines using single embedding tables or category-based filtering.
Online A/B Test: Deployed on the JD e-commerce search engine, the system showed significant improvements over the online baseline:
- User Click-Through Rate (UCTR): +0.42%
- User Add-to-Cart Rate (UACR): +0.37%
- User Conversion Rate (UCVR): +0.35%
- The authors note that given the platform's massive traffic, these lifts translate to substantial revenue growth.

5. Significance and Claims

The paper claims that MMRM successfully addresses the limitations of current multimodal ranking systems by leveraging the heterogeneity of e-commerce signals. By moving away from single-signal fine-tuning and single-embedding downstream modeling, MMRM provides a more comprehensive understanding of user intent. The successful deployment on JD.com, serving millions of daily users, validates the practical efficacy and scalability of the proposed framework in a real-world, high-stakes environment. The work underscores the value of "multiplex" representations in disentangling complex, multi-objective user behaviors.