Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models

1. Problem Statement

Reward Models (RMs) are essential for aligning Large Language Models (LLMs) via Reinforcement Learning from Human Feedback (RLHF). However, existing approaches face a fundamental trade-off between efficiency and accuracy:

• Scalar Reward Models (SRMs): These act as discriminators, outputting a single scalar score. They are computationally efficient (fast inference) but often lack the reasoning depth required for complex, out-of-distribution, or subtle evaluation tasks.
• Generative Reward Models (GRMs): These act as judges, generating Chain-of-Thought (CoT) reasoning before making a decision. While they achieve higher accuracy by mimicking human deliberation, they incur massive computational costs (generating hundreds of tokens) and are inefficient for high-throughput scenarios.

Current systems treat SRMs and GRMs as mutually exclusive alternatives. The paper argues that this is suboptimal, as human cognition (Dual Process Theory) utilizes both "fast" intuitive judgments for simple tasks and "slow" analytical reasoning for complex ones. The core research question is: Can we efficiently integrate SRMs and GRMs into a single model to achieve the best of both worlds?

2. Methodology: Fast-Slow Thinking Reward Model (F/S-RM)

The authors propose F/S-RM, a hybrid architecture that unifies scalar and generative paradigms within a single base model. The system operates on an adaptive reasoning chain, dynamically switching between "Fast" and "Slow" modes based on the model's internal confidence.

A. Architecture and Workflow

1. Fast Thinking (System 1): The model attempts to make a preference judgment immediately by predicting the first token (e.g., 'A', 'B', or 'Tie'). This mimics an SRM, requiring only a single forward pass.
2. Dual-Confidence Activation: The model calculates two confidence metrics after the first token prediction:
   • Intuition Confidence ($C_I$): The probability gap between the two candidate labels ($|P(A) - P(B)|$). A small gap indicates ambiguity.
   • Token Confidence ($C_T$): A measure of probability leakage to irrelevant tokens (entropy). High leakage indicates uncertainty.
   • Activation Rule: If both $C_I$ and $C_T$ fall below specific thresholds, the model triggers "Slow Thinking." Otherwise, it terminates with the fast judgment.
3. Slow Thinking (System 2): If triggered, the model appends a special "tie" token and generates a full Chain-of-Thought (CoT) reasoning trace before outputting the final judgment.

B. Two-Stage Progressive Training Framework

To prevent "action space collapse" (where the model fails to learn the conditional logic) and reward hacking, the authors employ a specific training pipeline:

1. Stage 1: Fast Thinking (Supervised Fine-Tuning - SFT):
   • The model is trained to predict the correct preference label ('A' or 'B') as the first token.
   • Loss Function: Combines the standard Bradley-Terry (BT) preference loss with an Action-Space Constraint to force probability concentration on the valid labels, preventing the model from generating irrelevant tokens during the fast phase.
2. Stage 2: Slow Thinking (Reinforcement Learning - RL):
   • Building upon the fast-thinking model, the authors use Group Relative Policy Optimization (GRPO).
   • Trigger Mechanism: A special "tie" token is appended to the prompt to instruct the model to bypass fast judgment and enter the CoT reasoning mode.
   • Reward Design: Includes a format reward (penalizing deviation from the two-stage structure) and a binary outcome reward (correctness of the final judgment).
3. Stage 3: Dual-Confidence Integration:
   • The final model ($\pi^*_\theta$) combines the capabilities of both stages. The dual-threshold mechanism determines the routing strategy during inference, optimizing the trade-off between accuracy and token consumption.

3. Key Contributions

• Hybrid Architecture: Introduced F/S-RM, the first RM to unify scalar (first-token) and generative (CoT) paradigms in a single model, inspired by Dual Process Theory.
• Dual-Confidence Activation: Designed a novel routing mechanism using two complementary confidence metrics (Intuition and Token confidence) to dynamically decide when to invoke expensive reasoning, ensuring efficiency without sacrificing accuracy on hard cases.
• Progressive Training Strategy: Developed a two-stage training framework (SFT for fast judgment + RL for slow reasoning) that stabilizes the learning of conditional reasoning paths and prevents performance degradation in the fast mode.

4. Experimental Results

The authors evaluated F/S-RM on three major benchmarks: RewardBench, RM-Bench, and JudgeBench, using Qwen3-4B and Qwen3-8B as base models.

• Performance vs. Efficiency:
  • The Hybrid F/S-RM achieved a 1.2% relative performance improvement over state-of-the-art models (specifically outperforming strong baselines like BR-RM and Skywork-Reward-V2 on complex tasks).
  • It reduced token consumption by 20.8% on average compared to full slow-thinking (GRM) baselines.
  • On the challenging JudgeBench, the hybrid model actually improved performance by ~0.57 points over the pure slow mode, demonstrating that the adaptive mechanism prevents overthinking on solvable tasks.
• Ablation Studies:
  • Training Stability: The SFT stage is crucial; training slow thinking without prior fast-thinking SFT led to performance collapse on difficult benchmarks (e.g., JudgeBench accuracy dropped from 69.4% to 55.1% on the 4B model).
  • Routing Efficacy: The dual-confidence mechanism outperformed single-metric routing strategies, achieving the best balance between accuracy retention and token savings.
  • Domain Adaptation: The model correctly identified that "Safety" tasks are mostly fast (83.4% fast rate), while "Math" and "Code" tasks require slow thinking, effectively allocating resources where needed.

5. Significance

This work represents a significant step forward in efficient RLHF. By moving away from the binary choice of "fast but dumb" vs. "slow and smart," F/S-RM demonstrates that adaptive inference is possible within reward modeling.

• Scalability: It makes high-accuracy reward modeling feasible for online, real-time applications where token cost is a bottleneck.
• Cognitive Alignment: It bridges the gap between LLM engineering and cognitive science, validating that Dual Process Theory can be effectively implemented in generative AI architectures.
• Future Direction: It suggests a new paradigm for LLM evaluation and alignment where models dynamically scale their computational effort based on task difficulty, potentially reducing the carbon footprint and latency of LLM deployment.