AdaRubric: Task-Adaptive Rubrics for LLM Agent Evaluation

1. Problem Statement

Current evaluation methods for Large Language Model (LLM) agents suffer from a "static-rubric bottleneck."

• Inadequacy of Fixed Metrics: Traditional metrics like ROUGE-L or BERTScore measure surface-level overlap and fail to capture goal-directed reasoning.
• Limitations of "LLM-as-Judge": Existing LLM-based evaluators (e.g., Prometheus, G-Eval) rely on fixed, pre-defined dimensions (e.g., Helpfulness, Fluency, Safety). These dimensions are designed for chat assistants, not for complex agents performing multi-step tasks like code debugging, API orchestration, or web navigation.
• The Consequence: A fixed rubric introduces systematic bias. For example, a "fluency" dimension might penalize an agent that correctly executes a complex API chain but produces compact, machine-readable output. Conversely, a generic "helpfulness" rubric fails to capture critical dimensions like "Error Recovery" or "Parameter Correctness" specific to tool-use tasks. This leads to weak correlation with human expert judgments and poor training signals for Reinforcement Learning (RL).

2. Methodology: The ADARUBRIC Framework

ADARUBRIC addresses these issues by generating task-specific evaluation rubrics on the fly from task descriptions. The framework operates through a three-stage pipeline followed by reward synthesis (see Figure 2 in the paper):

Stage 1: Adaptive Rubric Generation

• Input: A task description $T$ (instruction, domain, context, tools).
• Process: An LLM is prompted to generate a dynamic rubric $R(T)$ consisting of $N$ orthogonal evaluation dimensions (default $N=5$).
• Output: Each dimension includes:
  • A name (e.g., "Code Efficiency" for debugging).
  • A relative importance weight ($w_j$).
  • Calibrated 5-point verbalized scoring criteria ($\Gamma_j$) with concrete examples.
• Optimization: Rubrics are cached per task type to reduce API costs by >95%. Automated validation ensures dimensions are non-overlapping and weights sum to 1.

Stage 2: Confidence-Weighted Trajectory Evaluation

• Granular Scoring: For every step $k$ in an agent trajectory $\tau$, the evaluator LLM scores the step against each dimension $j$.
• Dual Output: The model returns a score $s_{k,j} \in \{1..5\}$ and a confidence weight $c_{k,j} \in [0,1]$. The confidence is low if the step does not directly engage the specific dimension (e.g., a reasoning step might have low confidence for a "Tool Accuracy" dimension).
• Aggregation: Step scores are aggregated into global dimension scores using strategies like Weighted Mean (WM) (default, up-weighting recent steps), Geometric Mean (GM), or Min Score (for safety-critical tasks).

Stage 3: Confidence-Filtered Selection & Reward Synthesis

• DimensionAwareFilter: A novel filtering mechanism that prevents "masking." A trajectory cannot pass if any single critical dimension falls below a threshold, even if the total score is high. This ensures no dimension-level failure is hidden by high scores in other areas.
• DPO Pair Construction: Filtered trajectories are sorted by their total score. Preference pairs $(\tau_{chosen}, \tau_{rejected})$ are constructed with a margin constraint ($S(\tau_i) - S(\tau_j) \ge \delta_{min}$).
• Reward Signal: These pairs serve as high-quality training data for Direct Preference Optimization (DPO) or as dense reward signals for PPO.

3. Key Contributions

1. Task-Adaptive Rubric Generation: The first framework to dynamically generate orthogonal, calibrated evaluation dimensions from task descriptions, eliminating the need for manual rubric engineering.
2. Multi-Dimensional Dense Rewards: Introduces a step-level, dimension-level scoring mechanism with confidence weighting, enabling fine-grained credit assignment for RL/DPO.
3. DimensionAwareFilter: A provably necessary condition to prevent high-scoring dimensions from masking failures in specific critical dimensions, ensuring robust preference pair selection.
4. Reliability Quantification: Establishes Krippendorff's $\alpha$ (target $\ge 0.80$) as a principled deployment criterion for LLM-based evaluators, ensuring consistency across evaluation runs.
5. End-to-End Pipeline: Demonstrates a complete workflow from evaluation to training, showing that adaptive rubrics significantly improve downstream agent performance without changing training infrastructure.

4. Experimental Results

The framework was evaluated on WebArena, ToolBench, and AgentBench, with generalization tested on SWE-bench.

• Human Correlation: ADARUBRIC achieved a Pearson correlation of $r=0.79$ with human expert rankings on WebArena, a significant improvement of +0.15 over the best static baseline (GPT-4 Direct, $r=0.64$).
• Reliability: Achieved Krippendorff's $\alpha = 0.83$, meeting the deployment criterion for high-reliability evaluation.
• Downstream DPO Training:
  • Agents trained on ADARUBRIC preference pairs showed +6.8 to +8.5 percentage points (pp) improvement in task success rates compared to Prometheus-trained agents.
  • On SWE-bench (code repair), ADARUBRIC generalized without any rubric engineering, achieving $r=0.77$ and a +4.9 pp gain in resolve rate over GPT-4 Direct.
• PPO Convergence: When used as a reward function for PPO, ADARUBRIC accelerated convergence, achieving +6.6 pp higher success rates at 5,000 steps compared to scalar reward baselines.
• Ablation Studies:
  • Adaptivity is Key: The gap between static and adaptive rubrics (+0.15) was larger than the gap between different backbone models (GPT-4o vs. Llama-3.1-8B), proving that rubric design is the primary driver of performance.
  • Optimal Dimensions: Performance peaked at $N=5$ dimensions; too few under-covered tasks, while too many introduced redundancy.

5. Significance and Impact

• Bridging the Evaluation Gap: ADARUBRIC solves the fundamental mismatch between generic chat-assistant evaluation criteria and the specific needs of goal-directed agents.
• Scalability & Automation: By automating rubric generation, it removes the bottleneck of manual dimension engineering, allowing evaluation systems to scale to new domains (e.g., from web navigation to code repair) instantly.
• Training Efficiency: The generation of high-quality, dense reward signals significantly reduces the sample complexity of RL training, leading to faster convergence and better final performance.
• Generalization: The framework's ability to transfer across domains (e.g., WebArena to SWE-bench) suggests that learning how to evaluate goal-directed reasoning is a transferable skill, independent of the specific task domain.

In conclusion, ADARUBRIC represents a paradigm shift from static, one-size-fits-all evaluation to dynamic, task-aware assessment, providing a robust foundation for the next generation of reliable and capable LLM agents.