Technical Summary: Can Coding Agents Build Robust Baselines?

Problem Statement

Developing competitive deep learning baselines for medical imaging remains a highly iterative, expertise-intensive process. It requires a complex sequence of interdependent decisions spanning preprocessing, architecture selection, optimization, data augmentation, loss design, and evaluation. These tasks demand both clinical and machine learning domain knowledge, making baseline development costly, time-consuming, and difficult to scale. While previous automation efforts have addressed isolated stages—such as hyperparameter optimization, neural architecture search, or automated data augmentation—they often fail to integrate these components into a unified workflow tailored for medical imaging. Furthermore, existing Large Language Model (LLM) based agents typically function as general-purpose research assistants rather than specialized pipelines for end-to-end baseline generation.

Methodology

The authors propose an agentic AI Scientist workflow that integrates literature-guided planning, automated implementation, and hypothesis-driven experimentation. The framework is built upon a "skill-based" approach where independent LLM-driven skills with explicit inputs and outputs orchestrate the development process. The workflow consists of three sequential pipelines:

1. SOTA Knowledge Pipeline: This pipeline transforms a raw dataset into an initial development plan. It performs a dual analysis from clinical and machine learning perspectives to identify key challenges and dataset characteristics. It synthesizes relevant literature and challenge reports to compare evidence across design axes (architecture, optimization, augmentation, loss functions). Uncertain or conflicting recommendations are formulated as experimental hypotheses. The pipeline also inspects code repositories to identify reusable components versus those requiring implementation.
2. Model Setup Pipeline: Based on the development plan, this pipeline configures the execution environment and implements the specific data loading, preprocessing, model architecture, optimization strategy, loss function, and evaluation metrics. It integrates these components into a shared PyTorch Lightning framework. Notably, data augmentation policies are excluded here, as they are optimized during the subsequent experimentation phase.
3. Experimentation Pipeline: This pipeline iteratively refines the baseline through evidence-driven experimentation. Rather than searching an unrestricted hyperparameter space, it follows a scientific workflow:
   • Hypothesis Generation: It defines evaluation criteria and tests hypotheses, prioritizing literature-supported alternatives.
   • Structured Execution: Rule-based logic selects the next component to evaluate, and reusable templates define the experiment structure.
   • Evidence Tracking: A persistent experiment tree records both positive and negative evidence. Rejected experiments are preserved as negative evidence to guide future decisions, preventing the agent from revisiting unproductive directions.
   • Constraints: The pipeline estimates clinically plausible parameter ranges for data augmentation using image-based inspection and literature constraints to ensure realistic variations.

The entire process is orchestrated using Claude Code (specifically the Claude Sonnet 4 model), with skills implemented as task-specific prompt templates exchanging structured Markdown, YAML, and CSV artifacts. Human supervision is maintained through lightweight review checkpoints where a researcher validates artifacts and launches suggested experiments, though manual code modifications were not required in the reported experiments.

Key Contributions

• Unified Agentic Workflow: The paper presents a novel framework that combines literature review, code generation, and hypothesis-driven experimentation into a single, cohesive pipeline for medical imaging, moving beyond isolated automation tasks.
• Skill-Based Architecture: The system decomposes the development process into specialized, reusable skills (Data Description, Data Preparation, SOTA Knowledge, Model Setup, Experimentation) that operate on structured artifacts, ensuring transparency and reproducibility.
• Persistent Experiment Tree: Unlike standard experiment tracking tools (e.g., MLflow, Weights & Biases) that primarily record metadata, this workflow maintains a structured tree of both accepted and rejected experimental evidence to actively inform future decision-making.
• Generalization Across Tasks: The workflow is designed to adapt to heterogeneous tasks (segmentation, classification, detection) without requiring task-specific redesign of the underlying agent logic.

Results

The framework was evaluated on four public medical imaging benchmarks: PUMA (tissue segmentation and nuclei detection), MILK10k (classification), and MIDOG25 (atypical mitosis detection).

• Performance Gains: Across all tasks, the Experimentation Pipeline consistently improved validation performance. The largest gains were observed during the "experiment exploration" phase, particularly through architecture exploration.
• Leaderboard Rankings:
  • PUMA: Achieved 6th place out of 15 teams in both the tissue segmentation and nuclei detection tracks.
  • MILK10k: Achieved 31st place out of 125 submissions (among methods not using external datasets).
  • MIDOG25: The resulting model demonstrated strong domain generalization across different scanners, tumor types, and species, achieving a validation F1 of 0.855 and a held-out test F1 of 0.847.
• Efficiency: The workflow produced competitive baselines within 2–7 days (measured from the first training run to the final selection), significantly reducing engineering effort.
• Refinement Process: The experiment tree successfully guided the agent to revise initial literature-based decisions (e.g., changing model architectures) while validating others (e.g., optimization strategies). For instance, on MIDOG25, the validation F1 improved from 0.757 to 0.855 through 30 experiments, with 17 accepted and 13 rejected.

Significance and Claims

The authors claim that skill-based, literature-guided agentic workflows can substantially reduce the engineering effort required to develop competitive medical imaging baselines while maintaining high performance. The study demonstrates that such workflows can:

1. Automate the majority of the ML development lifecycle, shifting the researcher's role from implementation to supervision and scientific reasoning.
2. Produce robust, reproducible baselines that generalize across diverse medical imaging domains without requiring dedicated domain-adaptation strategies.
3. Serve as a practical path toward scalable medical imaging research, allowing experts to focus on hypothesis generation and clinical interpretation.

The paper modestly notes that it evaluates the integrated workflow rather than isolating individual components via ablation studies. It acknowledges limitations such as the use of a single random seed, the lack of comparison against human-developed baselines under identical budgets, and the current inability of the agent to autonomously redesign the workflow or revisit literature in response to unexpected findings. Nevertheless, the results suggest that structured AI Scientist workflows are a viable approach for automating complex, iterative scientific processes in medical imaging.