Technical Summary: Pailitao-MMSearch

Problem Statement

The evolution of e-commerce search has shifted from simple text-based queries to complex multimodal interactions involving images, natural language, and mixed-intent instructions. However, existing production systems face a critical dilemma:

1. Single-modal Specialist Models: Deployed independently for text, visual, and voice tasks, these models operate in isolated representation spaces. They cannot natively handle cross-modal queries where meaning emerges from the interaction between modalities, forcing systems to rely on ad-hoc fusion heuristics that increase latency and complexity.
2. General-Purpose Vision-Language Models (VLMs): While capable of processing images and text, these models lack critical e-commerce domain knowledge. They fail to understand fine-grained product attributes (e.g., material composition, brand tiers), lack collaborative filtering signals (user preferences, co-occurrence patterns), and cannot directly generate structured product identifiers for retrieval from billions of items.

These limitations result in user experience failures, such as ignoring textual constraints in favor of image content or failing to capture true commercial intent.

Methodology

The authors propose Pailitao-MMSearch, a native e-commerce multimodal search foundation model built upon the Qwen3 architecture and Qwen2-VL vision encoder. The approach reformulates multimodal search as an end-to-end autoregressive product generation problem, consisting of three core innovations:

1. HybSID (Hybrid Semantic ID)

To enable generative retrieval over a billion-scale catalog while maintaining fine-grained discrimination, the authors introduce a novel product tokenization scheme:

• Discrete Semantic Codes: Products are encoded into a sequence of three discrete tokens ($c_1, c_2, c_3$) using Residual Quantization (RQ-VAE). These provide coarse-to-fine categorical grouping for efficient autoregressive generation.
• Continuous Multimodal Embedding: To recover the fine-grained visual and attribute details lost during quantization, each product retains a continuous embedding ($e$).
• Unified Representation: A special token <|emb_token|> is introduced. In the generation phase, the model autoregressively produces the discrete codes followed by <|emb_token|>, whose hidden state is decoded via a lightweight MLP (HybDecoder) to reconstruct the continuous embedding. This allows the model to perform both coarse retrieval (via SID) and fine-grained ranking (via embedding similarity) within a single framework.

2. Two-Stage Continual Pre-training (CPT)

To inject domain knowledge without catastrophic forgetting, the authors employ a two-stage pipeline:

• Stage 1 (Domain Knowledge Injection): The model is trained on e-commerce specific objectives, including Product Knowledge Alignment (mapping between HybSID and product titles/attributes) and User Behavior Alignment (predicting next interactions and modeling co-occurrence patterns). A small proportion of general natural language data is included to act as a regularizer.
• Stage 2 (Capability Preservation via On-Policy Distillation): Since the original pre-training corpus is inaccessible, the authors use the base model as a teacher. They sample responses on-policy from the Stage 1 student model and minimize the Kullback-Leibler (KL) divergence between the student's output distribution and the teacher's distribution on general-domain prompts. This recovers instruction-following and reasoning capabilities lost during domain adaptation.

3. Hybrid Reasoning Post-Training Pipeline

The final model is adapted for diverse search intents through a multi-phase pipeline:

• Multi-Task SFT: Data is constructed from real user trajectories, stratified by difficulty.
• Hybrid Reasoning: The model is trained to use direct generation for simple queries (normal difficulty) and Chain-of-Thought (CoT) reasoning for complex intents (hard difficulty), decomposing the search process into explicit steps (attribute identification, intent parsing, reasoning, generation).
• Reinforcement Learning (RL): Using Group Relative Policy Optimization (GRPO), the model is optimized with verifiable rewards based on SID accuracy and embedding similarity against a curated set of ground-truth products, rather than single logged clicks.
• Multi-Expert On-Policy Distillation: The final model fuses capabilities from specialized RL expert models, the base LLM, and the SFT model into a unified system.

Key Contributions

1. HybSID Tokenization: A hybrid scheme combining discrete semantic IDs with continuous multimodal embeddings, enabling fine-grained product understanding and end-to-end autoregressive generation within a single language model.
2. Two-Stage CPT Strategy: A method to inject e-commerce domain knowledge (product attributes and user behavior) while mitigating catastrophic forgetting through On-Policy Distillation, even without access to the original pre-training corpus.
3. Hybrid Reasoning Pipeline: A post-training framework integrating difficulty-aware CoT reasoning, verifiable reward-based RL, and multi-expert knowledge fusion to handle both simple and complex multimodal search intents.
4. Industrial Deployment: The model is deployed on Taobao's Pailitao platform, serving tens of millions of daily active users.

Results

The paper reports extensive online A/B testing and offline evaluations:

• Online Performance: On the "PeiWoGuang" floor (a visual search scenario), the combined generative retrieval and ranking deployment achieved a +13.61% increase in Gross Merchandise Volume (GMV) and a +8.21% increase in Transaction Volume compared to the traditional multi-modal pipeline. Overall Pailitao platform metrics also showed positive lifts (+1.02% GMV, +0.53% Transaction Volume).
• Offline Evaluation:
  • Domain Understanding: The model demonstrated strong capability in translating SIDs to captions and attributes, and in multimodal SID recognition (End-to-end Retrieval@10 reached 4.43%).
  • General Language Abilities: The Stage 2 CPT model successfully recovered instruction-following capabilities (IFEval strict-prompt accuracy improved from 36.78% to 38.45% compared to Stage 1), while maintaining performance on general benchmarks like MMLU-Redux and C-Eval.
  • Instruction Following: In multimodal search tasks, the model achieved high Strict Instruction Following (SIF) rates (e.g., 55.0% for Matching Product Search) and strong Recall@10 (0.8544).

Significance and Claims

The authors claim that Pailitao-MMSearch addresses the fundamental limitations of both single-modal specialist systems and general-purpose VLMs by creating a native e-commerce multimodal search foundation.

• Native Integration: Unlike approaches that rely on external retrieval modules or tool-calling, this model tokenizes the entire product catalog within its vocabulary, enabling true end-to-end generation.
• Domain-Native Foundation: The paper argues that the critical bottleneck for future agentic search systems is not the orchestration framework, but the availability of a foundation model with genuine e-commerce domain understanding. Pailitao-MMSearch serves as this essential infrastructure.
• Practical Effectiveness: The substantial improvements in GMV and transaction volume validate that integrating domain knowledge, user behavior modeling, and multimodal reasoning into a single generative model yields superior commercial outcomes compared to traditional cascaded architectures.

The authors acknowledge current limitations, particularly in handling highly complex, noisy, or ambiguous multimodal inputs (e.g., low-resolution images with typos), and identify enhancing multimodal reasoning capabilities as a priority for future iterations.