Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning

1. Problem Statement

The paper addresses the challenge of simulating real human behavior in web-based online shopping environments using Large Language Models (LLMs).

• The Gap: Existing approaches primarily rely on Supervised Fine-Tuning (SFT) using synthetic data (rationales generated by other LLMs) or zero-shot prompting. These methods are limited because:
  • They are bounded by the reasoning capabilities of the model used to generate the training data.
  • They struggle with long-horizon sequences and high-entropy free-form text (e.g., specific button names, search queries).
  • They lack the ability to adapt to the non-deterministic nature of human decision-making, where multiple valid actions exist.
• The Objective: To develop a framework that moves beyond simple task completion (where the goal is fixed) to behavior simulation, where the model must learn the distribution of human actions and the underlying rationales based on observable web contexts (HTML) and action history.

2. Methodology: Shop-R1 Framework

The authors propose Shop-R1, a Reinforcement Learning (RL) framework that decomposes the simulation task into two distinct stages: Rationale Generation and Action Prediction. The framework utilizes a hybrid reward design to guide policy optimization.

A. Training Pipeline

1. Cold Start (SFT): The model (Qwen-2.5-3B-Instruct) is first fine-tuned on trajectories containing rationales generated by Claude 3.5 Sonnet. This grounds the model in realistic action patterns and structural dependencies before RL begins.
2. RL Optimization: The model is further optimized using Group Relative Policy Optimization (GRPO). Crucially, no LLM-generated rationales are used during the RL stage; the model learns to generate its own rationales and actions based on the reward signals.

B. Reward Design (The Core Innovation)

Shop-R1 introduces a multi-component reward system to prevent "reward hacking" (e.g., the model always choosing the easiest action) and to encourage fine-grained accuracy:

1. Format Reward (Binary):

   • Awards 0.5 only if the output is a valid JSON structure containing rationale and action keys.
   • Ensures parseability and provides a baseline signal for structural correctness.

2. Rationale Reward (Self-Certainty):

   • Since ground-truth human rationales are unavailable, the system uses a self-certainty signal.
   • It calculates the KL Divergence between the model's output token distribution and a uniform distribution.
   • Higher certainty (lower entropy) in the generated rationale yields a higher reward, acting as a self-supervised signal for reasoning quality.

3. Hierarchical Action Reward with Difficulty-Aware Scaling (DARS):

   • Instead of a binary "correct/incorrect" signal, rewards are assigned hierarchically:
     • Action Type Reward: Correctly identifying the high-level type (e.g., click, type_and_submit, terminate).
     • Sub-action Attribute Reward: Correctly identifying specific attributes (e.g., the button name).
     • Value Reward: Correctly generating long-text values (e.g., search query text), evaluated via ROUGE-L similarity.
   • Difficulty-Aware Scaling (DARS): To prevent the model from exploiting easy actions (like terminate), rewards for complex actions (like type_and_submit) are scaled up by a factor (default 1000) based on their difficulty. This makes the "easy path" less lucrative than the "hard path."

Objective Function:
The model maximizes a combined objective:
$$\max_{\pi_\theta} E_{r,a \sim \pi_\theta(q)} [v(a) + \alpha s(r) - \beta KL(\pi_\theta \parallel \pi_{ref})]$$
Where $v(a)$ is the hierarchical action reward, $s(r)$ is the self-certainty score, and $KL$ is a regularization term against the reference policy.

3. Key Contributions

1. First RL Framework for Web-Shopping Simulation: The paper is the first to apply RL specifically to simulation-oriented human behavior modeling in e-commerce, moving beyond task-completion agents.
2. Two-Stage Decomposition: It reformulates the problem into rationale generation and action prediction, assigning distinct reward signals to each to improve interpretability and alignment.
3. Hybrid Reward Architecture:
   • Introduces Self-Certainty for rationale generation (overcoming the lack of ground-truth rationales).
   • Proposes a Hierarchical Reward with Difficulty-Aware Scaling to solve the reward hacking problem and encourage the learning of complex, high-entropy behaviors.
4. Comprehensive Empirical Validation: Extensive ablation studies and comparisons across different model sizes (0.5B, 1.5B, 3B) demonstrate the necessity of each component.

4. Experimental Results

Experiments were conducted on the SHOP-CART dataset (52,137 real-world sessions) using the Qwen-2.5-3B-Instruct model.

• Performance Gains:
  • Shop-R1 achieved an Exact Match Accuracy of 27.72%.
  • This represents a >65% relative improvement over the best baseline (SFT-only, which achieved 16.76%).
  • It also significantly outperformed Zero-shot prompting (0.32%) and SFT + Binary RL (16.55%).
• Action Type Analysis:
  • Shop-R1 improved accuracy across all action types, particularly for complex actions like type_and_submit (7.53% vs. 3.84% for SFT).
  • It successfully reduced the bias toward the trivial terminate action, which plagued other methods.
• Ablation Insights:
  • SFT Warm-start: Removing SFT caused exact match accuracy to drop to 4.63%, proving the need for supervised priors.
  • Format Reward: Removing this caused accuracy to plummet to 2.87% due to unparseable outputs.
  • DARS: Removing difficulty scaling led to a drop in F1 scores (from 31.28% to ~12%), as the model reverted to "reward hacking" by selecting only easy actions.
  • Context: Using full session history (HTML) was critical; using only the latest step reduced accuracy by nearly 50%.

5. Significance

• High-Fidelity Simulation: Shop-R1 demonstrates that LLMs can be trained to simulate realistic, nuanced human behaviors in complex web environments, not just complete predefined tasks.
• Scalable Reward Design: The proposed reward mechanism (Self-Certainty + Hierarchical DARS) offers a blueprint for training RL agents in domains where ground-truth labels for reasoning are unavailable and where action spaces vary in complexity.
• Practical Applications: This technology enables the creation of realistic virtual users for A/B testing, user experience (UX) research, and e-commerce system evaluation without relying on expensive human data collection.
• Generalizability: While focused on shopping, the framework's design (handling GUI interactions, HTML context, and hierarchical rewards) is applicable to other web-based agent tasks.

In conclusion, Shop-R1 establishes a new state-of-the-art for human behavior simulation by effectively combining supervised initialization with a sophisticated, multi-faceted reinforcement learning strategy that rewards both the intent and the precision of human-like actions.