Technical Summary: MORE – Adaptive Multi-Objective Learning for E-commerce Dialogue Systems

Problem Statement

In e-commerce dialogue systems, there exists a fundamental tension between two critical objectives: reasoning correctness and linguistic naturalness. Systems must accurately reason over structured user profiles (e.g., credit limits, eligibility, purchase history) to ensure factual accuracy and trustworthy decision-making, while simultaneously generating responses that are fluent, natural, and engaging.

The paper identifies that these goals are complementary but not identical. Existing approaches often fail to balance them:

• Rule-based or symbolic systems guarantee correctness but produce rigid, unnatural utterances.
• Personalized dialogue models improve persona consistency but often neglect the factual correctness of dynamic user attributes.
• Standard Reinforcement Learning (RL) approaches that naively mix multiple reward signals (e.g., summing reasoning accuracy and generation quality) lead to unstable optimization dynamics. As demonstrated in the paper's preliminary experiments, directly combining heterogeneous rewards causes significant training oscillations and variance, often degrading both reasoning accuracy and response quality compared to optimizing them individually.

Methodology: The MORE Framework

The authors propose MORE (Multi-Objective REinforcement learning), an adaptive framework designed to jointly optimize reasoning accuracy and linguistic naturalness without the instability of direct reward mixing. The framework consists of three core components:

1. Reasoning-Enhanced Training Scaffold

Instead of jointly optimizing reasoning and response generation in a single step (which causes conflicting gradients), MORE treats reasoning as a training scaffold.

• Profile Information Reasoning: The model is explicitly trained on two types of tasks using a profile database:
  • Masked Attribute Completion (MAC): Recovering specific attributes (e.g., last four digits of a phone number) from natural language contexts.
  • Programmatic Numerical Reasoning (PNR): Performing numerical inference based on attributes (e.g., calculating daily interest).
• Scaffold Objective: These tasks are optimized via Reinforcement Learning with verifiable rewards (1 for correct, 0 for incorrect). This stage enhances the model's ability to infer user attributes and perform factual recall.
• Inference Efficiency: Crucially, during inference, the model does not generate explicit reasoning steps. It directly generates the final response, leveraging the reasoning capabilities learned during the scaffold phase to maintain low latency and output length.

2. Adaptive Multi-Reward Optimization

To handle the generation phase, MORE employs an adaptive mechanism to balance multiple, potentially conflicting objectives (fluency, naturalness, semantic fidelity, coverage, etc.).

• Multi-Dimensional Evaluation: Candidate responses are evaluated across multiple dimensions using LLM-based reward models (e.g., DeepSeek-R1) and traditional metrics (ROUGE).
• Gradient-Based Dynamic Reweighting: Rather than using fixed weights, MORE employs a rank-based linear decay strategy to dynamically adjust reward weights based on gradient feedback.
  • The magnitude of the gradient contribution for each reward is calculated.
  • Weights are assigned based on the rank of these gradients rather than their absolute magnitude. This approach stabilizes training by discarding unreliable magnitude information while preserving consistent ordering, preventing the model from being overly sensitive to small numerical fluctuations or dominated by a single strong signal.
• Reference Model: The reasoning-enhanced model serves as a reference ($\pi_{ref}$) to anchor correctness, while the policy model focuses on improving naturalness within the GRPO (Group-wise Reinforcement Policy Optimization) framework.

3. Training Pipeline

The training process (Algorithm 1) involves:

1. Updating the policy on MAC/PNR tasks to create a reasoning-anchored reference model.
2. Generating a group of candidate responses.
3. Computing a set of diverse rewards for each candidate.
4. Dynamically reweighting these rewards based on gradient ranks.
5. Optimizing the policy using a PPO-style objective with the aggregated, adaptive reward and a KL-divergence penalty against the reference model.

Key Contributions

1. Identification of the Multi-Goal Challenge: The paper highlights the specific instability caused by directly mixing reasoning and generation rewards in e-commerce dialogue systems, demonstrating that naive combination leads to oscillation and performance degradation.
2. MORE Framework: The proposal of an adaptive multi-objective RL framework that decouples reasoning (as a training scaffold) from generation, utilizing a gradient-based dynamic reweighting mechanism to balance competing objectives.
3. Empirical Validation: Comprehensive evaluation across offline benchmarks and real-world production environments, demonstrating that MORE outperforms strong baselines in both reasoning accuracy and conversational quality.

Experimental Results

Offline Evaluation

• ByteDance Datasets (Proactive Sale & Customer Service): MORE consistently outperformed baselines including SFT, DPO, KTO, RLHF, CoT, and other multi-objective RL methods (C-DynaOpt, EMORL). It achieved significant improvements in BLEU, ROUGE, Naturalness, Reasoning, Fidelity, and Coverage metrics.
• MultiWOZ 2.2 Benchmark: MORE achieved state-of-the-art performance in task success (Inform: 92.5%, Success: 84.8%) and lexical diversity, outperforming both full-shot training models and zero-shot prompting methods.

Online A/B Testing (ByteDance Production)

A 14-day online experiment was conducted on real production traffic:

• Proactive Sales: MORE improved Overall Conversion by 16.53% and Reached Conversion by 30.09% compared to the baseline.
• Customer Service: User satisfaction increased by 1.12%, and the handoff rate to human agents decreased by 1.83%.
• Human-Machine Comparison: In a direct comparison with human agents in proactive sales, the MORE system recovered approximately 60% of the incremental conversion lift achieved by human agents, while reaching 89% of the absolute conversion rate.

Efficiency

By omitting explicit reasoning steps during inference, MORE reduced average latency from 5.69s to 1.71s (69.9% reduction) and output length from 252.6 to 79.8 tokens (68.4% reduction) compared to Chain-of-Thought (CoT) baselines.

Significance and Claims

The paper claims that MORE successfully addresses the "one model, multiple goals" challenge in e-commerce dialogue systems. Its significance lies in:

• Stabilizing Multi-Objective Learning: By treating reasoning as a scaffold and using adaptive gradient-based reweighting, it avoids the training instability inherent in direct reward mixing.
• Practical Business Impact: The framework delivers measurable business value, significantly improving conversion rates and user satisfaction in large-scale, real-world deployments.
• Balancing Trust and Usability: It demonstrates that a system can be both factually trustworthy (via reasoning) and naturally engaging (via adaptive generation) without incurring the inference overhead of explicit reasoning steps.

The authors conclude that effectively balancing these multiple dialogue objectives leads to tangible improvements in both dialogue quality and business outcomes, offering a scalable solution for complex e-commerce scenarios.