AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research

Problem
The automation of scientific research using language model (LM) agents is a rapidly growing field, yet evaluating their ability to generate insightful scientific contributions remains a significant challenge. A critical mechanism for deriving scientific insight is the ablation experiment, which dissects a proposed method into its components to evaluate their individual contributions. While previous work has addressed the execution of research code or the generation of hypotheses, the specific task of planning ablation experiments—identifying key method components and proposing meaningful, feasible, and grounded changes—has not been fully explored in an automated setting. Existing benchmarks like AbGen rely on manually processed research contexts rather than raw paper content and focus solely on author-assisted planning, leaving a gap in evaluating agents that can assist both authors in planning experiments and reviewers in identifying missing validations.

Methodology
The authors introduce AblationBench, a benchmark suite designed to evaluate LM agents on ablation planning tasks in empirical AI research. The suite comprises two distinct tasks:

1. AuthorAblation: This task involves generating an ablation plan based on a paper's method section (excluding experimental results). The dataset contains 83 conference papers spanning 14 conferences, with 230 human-annotated ablation experiments serving as ground truth (GT).
2. ReviewerAblation: This task involves identifying missing ablation experiments in a full paper submission, simulating a reviewer's role. The dataset consists of 350 ICLR submissions (2023–2025) and their official reviews, where reviewers explicitly suggested missing ablations.

For both tasks, the authors developed LM-based judges to automate evaluation. These judges compare generated ablation plans against the GT (original author ablations for AuthorAblation; reviewer suggestions for ReviewerAblation) to determine semantic matches. To ensure robustness, the judges employ strategies to mitigate intra-model, contextual, and positional biases, such as using ensemble voting and randomizing input order.

The authors established baseline performance using two types of planners:

• LM-PLANNER: A single-step Chain-of-Thought (CoT) prompt to an LM.
• AGENT-PLANNER: A ReAct-style agent (based on SWE-agent) operating in a containerized environment with file inspection and execution tools.

Key Contributions

1. AblationBench: A comprehensive benchmark suite containing two tasks (AuthorAblation and ReviewerAblation) with 433 total instances, covering diverse AI domains (ML, CV, NLP) and providing a complete resource for developing and evaluating ablation agents.
2. Automated Evaluation Framework: Two LM-based judges (one for each task) capable of evaluating generated plans against ground truth, validated through a separate human-annotated dataset (JudgeEval) to ensure alignment with human judgment.
3. Baseline Performance Analysis: An evaluation of current frontier LMs (including GPT-4o, Claude 3.5 Sonnet, Llama 3.1, o3-mini, and Gemini 2.5 Flash) on ablation planning, revealing significant performance gaps compared to human experts.

Results
Experiments with frontier LMs demonstrate that ablation planning remains a challenging task:

• Performance: The best-performing LM system identified only 38% of the original ablations on average across tasks. On a subset of AuthorAblation tasks, the best model achieved an F1 score of 0.42, compared to 0.65 for human experts.
• Planner Architecture: Contrary to expectations for complex reasoning tasks, the LM-PLANNER (single CoT step) outperformed the AGENT-PLANNER (multi-step reasoning with tools) in both accuracy and cost (approximately 1.5x cheaper). The agent approach showed no correlation between trajectory length and performance.
• Task Differences: An inverse performance trend was observed between the two tasks. Models that performed well on AuthorAblation (requiring strict adherence to method descriptions) often performed poorly on ReviewerAblation (requiring creativity to identify gaps), and vice versa. This is attributed to differences in model grounding.
• Ablation Types: Models were significantly better at identifying "removal" ablations (removing a component) than "modification" ablations (replacing a component with a specific alternative), suggesting difficulty in proposing domain-specific substitutions.
• Human Baseline: Humans demonstrated substantially higher precision and recall than models, indicating that the benchmark is not yet saturated and that current models lack the nuanced understanding required for high-quality ablation planning.

Significance
The paper claims that AblationBench provides a necessary resource for advancing the field of AI co-scientists by isolating and evaluating the specific capability of ablation experiment design. The results highlight that while LMs can automate parts of the research process, they currently fall short of human-level performance in the critical step of experimental planning. The finding that simple CoT prompting outperforms complex agent-based approaches for this specific task suggests that current agent tools may not be well-aligned with the cognitive requirements of ablation planning. The authors position this work as a step toward developing stronger models capable of creative and grounded scientific contribution, while acknowledging limitations such as the reliance on proprietary models for judges and the difficulty of evaluating novel ablation ideas that do not exist in the ground truth.