Thinking Broad, Acting Fast: Latent Reasoning Distillation from Multi-Perspective Chain-of-Thought for E-Commerce Relevance

Technical Summary: Thinking Broad, Acting Fast

Problem Statement
Effective relevance modeling is fundamental to e-commerce search, yet traditional models struggle with long-tail and ambiguous queries. While Large Language Models (LLMs) enhanced with Chain-of-Thought (CoT) reasoning have shown promise in improving accuracy and interpretability, two critical limitations hinder their real-world deployment:

1. Single-Perspective Limitation: Existing CoT approaches typically rely on a single, monolithic reasoning perspective. This fails to capture the multifaceted nature of e-commerce relevance, which requires balancing user intent, structured attribute matching, and specific business rules simultaneously.
2. Distillation Inefficiency: Current knowledge distillation methods that utilize LLM-generated rationales often discard the CoT structure during inference. They treat reasoning as a transient training signal rather than an internal capability, forfeiting the utility of the reasoning process for online serving. Furthermore, LLMs themselves suffer from high inference latency, making them impractical for real-time search systems.

Methodology
The authors propose a framework that "thinks broad" during training and "acts fast" at inference, consisting of two primary components:

1. Multi-Perspective Chain-of-Thought (MPCoT) for Teacher Training
The framework employs a powerful teacher LLM (Qwen3-14B) trained to generate diverse rationales from three distinct perspectives:

• User Intent: Focuses on functional needs and use-case alignment.
• Structured Analysis: Performs systematic, attribute-by-attribute verification between query and product.
• Business Rules: Applies platform-specific heuristics to resolve ambiguities (e.g., distinguishing main products from accessories).

The teacher model undergoes a two-stage optimization process:

• Supervised Fine-Tuning (SFT): The model is trained on an aggregated dataset of high-quality, perspective-specific CoT rationales. This forces the model to internalize diverse reasoning paths.
• Direct Preference Optimization (DPO): To address cases where different perspectives yield conflicting predictions, the model is further refined using DPO. This stage constructs preference pairs where "chosen" responses come from correct perspectives and "rejected" responses from incorrect ones, teaching the model to adaptively weigh and select the optimal reasoning path based on context.

2. Latent Reasoning Knowledge Distillation (LRKD) for Student Deployment
To transfer the teacher's capabilities to a lightweight student model (e.g., BERT) without incurring generation latency, the authors introduce LRKD.

• Architecture: The student model is a cross-encoder augmented with a trainable Latent Reasoning Extractor. This extractor maps the student's contextualized token representations into a structured latent reasoning vector.
• Training Objective: The student is trained using a dual-loss function:
  1. Classification Loss: Standard cross-entropy for predicting relevance labels.
  2. Latent Guidance Loss: A Mean Squared Error (MSE) loss that aligns the student's latent reasoning vector with the dense embedding of the teacher's generated CoT (encoded by a frozen sentence embedding model like BGE-M3).
• Inference: Unlike prior methods, the latent reasoning extractor is retained during inference. It allows the student model to internalize sophisticated reasoning semantics in a compact, non-generative form, enabling low-latency deployment.

Key Contributions

1. MPCoT Framework: A novel reasoning framework that generates complementary rationales from multiple perspectives, demonstrating that fusing diverse viewpoints (User Intent, Structured Analysis, Business Rules) leads to more robust relevance judgments than single-perspective approaches.
2. LRKD Mechanism: A distillation technique that preserves reasoning semantics in a lightweight student model via a latent extractor. This avoids the high latency of text generation while retaining the utility of CoT reasoning during online serving.
3. Empirical Validation: Comprehensive validation across multilingual e-commerce datasets (AliExpress and ESCI) and a large-scale online A/B test.

Results

• Offline Performance: The combined MPCoT teacher and LRKD student achieved state-of-the-art results on the AliExpress and ESCI datasets. The MPCoT teacher outperformed the strongest single-perspective baselines by an average of +1.96% in Accuracy and +3.5% in F1. The LRKD student variants (specifically GAT and Poly-encoder based) consistently outperformed existing distillation baselines (CED-KD, MKD) and standard BERT models, with the GAT variant improving accuracy by +2.37 points and F1 by +3.91 points over the baseline.
• Efficiency: The student models maintained low latency (approx. 132–148 ms) compared to the teacher LLM (46,800 ms), with only marginal parameter increases.
• Online Impact: In a live A/B test on AliExpress's search advertising platform serving tens of millions of users, the method delivered statistically significant improvements:
  • +1.42% in Revenue Per Mille (RPM).
  • +0.48% in Click-Through Rate (CTR).
  • +0.4% in Relevance Satisfaction (RS) score.

Significance
The paper claims that its approach successfully bridges the gap between the high reasoning capabilities of LLMs and the strict latency requirements of industrial e-commerce search. By moving beyond single-perspective reasoning and retaining reasoning structures in the latent space of the student model, the framework offers a practical solution for handling complex, ambiguous queries that traditional models miss. The results demonstrate that "thinking broad" (via multi-perspective training) and "acting fast" (via latent distillation) can simultaneously enhance commercial performance (revenue) and user experience (relevance satisfaction) without the computational cost of running large LLMs in real-time.