Technical Summary: AdaJudge

Problem Statement

Reward modeling is critical for aligning Large Language Models (LLMs) with human preferences, yet current architectures face two fundamental structural limitations:

1. Static Inductive Bias: Predominant reward models rely on fixed pooling strategies (e.g., last-token or mean pooling) to condense sequence representations into scalar scores. This creates a mismatch with the task-dependent nature of preference signals. For instance, reasoning tasks often require localized terminal cues (favoring last-token pooling), while safety or coherence tasks require holistic, distributed evidence (favoring mean pooling). A single static strategy forces a trade-off, often failing to capture the spatial distribution of evidence required for specific domains.
2. Representational Mismatch: Backbone LLMs are optimized for next-token prediction (generative tasks), whereas reward modeling requires fine-grained pairwise discrimination. Consequently, the raw hidden states of the backbone may lack the necessary features to distinguish subtle preference cues, such as logical inconsistencies or minor constraint violations, without further adaptation.

Existing solutions often rely on scaling data or model size, or use computationally expensive ensembles of multiple models to achieve adaptability, leaving the fundamental architectural rigidity of single-model reward heads unaddressed.

Methodology: AdaJudge

The authors propose AdaJudge (Adaptive Multi-Perspective Judging), a unified two-stage framework designed to jointly adapt representation and aggregation without requiring token-level or process supervision.

Stage I: Adaptive Representation Refinement

To address the representational mismatch, AdaJudge introduces a lightweight iterative refinement module applied after the backbone LLM.

• Mechanism: The module consists of $K$ sequential transformer blocks.
• Depth-Gating: A global sequence-level context vector (derived via mean pooling of backbone features) is passed through a gating network to produce mixture coefficients $\alpha$.
• Refinement: The final refined representation $\tilde{H}$ is a convex combination of all intermediate hidden states ($H^{(1)}$ to $H^{(K)}$). This allows the model to dynamically emphasize shallow cues or accumulate deeper reasoning traces depending on the complexity of the input sample.

Stage II: Multi-Perspective Aggregation

To address the static inductive bias, AdaJudge replaces the fixed readout with a Domain-Aware Gated Mixture-of-Pooling head.

• Parallel Experts: The refined representation $\tilde{H}$ is processed through three parallel pooling experts:
  1. Last-Token Pooling ($z_L$): Captures conclusion-sensitive signals.
  2. Mean Pooling ($z_M$): Captures global stylistic and coherence attributes.
  3. Attention Pooling ($z_A$): Detects sparse anomalies anywhere in the sequence.
• Prompt-Conditioned Routing: A separate prompt representation ($z_P$) is derived via mean pooling of prompt tokens. A routing network takes the concatenated vector $[z_L; z_M; z_A; z_P]$ and outputs mixture weights $\pi$ via softmax.
• Dynamic Scoring: The final reward score $r(x, y)$ is a gated mixture of the scores from the three perspectives ($s_L, s_M, s_A$), weighted by $\pi$. This allows the model to adaptively select the aggregation strategy that best aligns with the spatial distribution of evidence required by the specific prompt.

Optimization

The model is trained using a Focal Bradley–Terry objective. The loss function includes a focal term to emphasize hard samples and an entropy regularization term to prevent the routing distribution from collapsing onto a single perspective, ensuring the model maintains a diverse set of aggregation strategies.

Key Contributions

1. Identification of Structural Mismatches: The paper formally identifies that generative-focused backbones lack fine-grained discrimination features and that fixed pooling fails to capture the multi-granular nature of preference evidence.
2. Unified Framework: The introduction of AdaJudge, which jointly adapts representational refinement (via depth-gated blocks) and evidence aggregation (via domain-aware routing) without external supervision.
3. Empirical Superiority: Demonstrated consistent outperformance over strong off-the-shelf reward models and controlled baselines (same backbone, different heads) across diverse domains and model scales.

Experimental Results

Experiments were conducted on RM-Bench and JudgeBench, utilizing base models Phi-3.5-mini-instruct, Qwen3-4B, and Qwen3-8B trained on the HelpSteer3 dataset.

• Performance Gains: AdaJudge significantly outperformed traditional pooling baselines. Notably, a Qwen3-8B model equipped with AdaJudge (71.1 overall on RM-Bench) surpassed a much larger 27B-parameter Skywork-Reward-Gemma-2 model (70.5), demonstrating high parameter efficiency.
• Resolution of Trade-offs: Static baselines exhibited conflicting performance profiles (e.g., last-token pooling excelled in safety but failed in math; mean pooling did the opposite). AdaJudge achieved a "Pareto improvement," simultaneously maximizing performance in Math (80.4), Safety (87.5), and robustness on hard subsets (43.0).
• Ablation Studies:
  • Removing the refinement stage resulted in significant drops in reasoning-intensive tasks (Math and Code), confirming the necessity of adapting the representation space.
  • Static single-view baselines (Last-Token, Mean, Attention) consistently underperformed the adaptive mixture, particularly on the "Hard" subsets of benchmarks.
• Internal Mechanism Analysis:
  • Alignment: Iterative refinement increased the cosine alignment between the scoring direction and the true preference difference, particularly for mean and attention pooling views.
  • Routing Behavior: The gating mechanism learned distinct aggregation patterns per domain (e.g., higher weight on Attention pooling for Code/Math, and Mean pooling for Safety), validating the model's ability to adapt to domain-specific evidence distributions.

Significance

The paper argues that the rigidity of fixed aggregation strategies is a primary bottleneck in reward modeling. By decoupling representation refinement from evidence aggregation, AdaJudge demonstrates that dynamic, instance-adaptive architectures can overcome the limitations of static inductive biases. The work suggests that future reward modeling should move beyond scaling data or model size alone, focusing instead on architectural flexibility that aligns internal representations and aggregation strategies with the specific spatial and logical demands of the evaluation task.

Limitations

The authors acknowledge several limitations:

• Scalability: Validation is currently limited to models up to 8B parameters; efficacy on extreme-scale models (70B+) remains unexplored.
• Latency: While lightweight, the iterative refinement and routing network introduce additional parameters and inference latency compared to standard fixed-pooling baselines.
• Expert Library: The current implementation uses a fixed set of three pooling experts. The framework is extensible, but the optimal set of experts and potential for automatic expert discovery are open questions.
• Modality: Evaluation is currently restricted to text-only preference modeling, with potential for extension to multimodal inputs.