Agentic systems for breast cancer treatment recommendations

Technical Summary: Agentic Systems for Breast Cancer Treatment Recommendations

Problem Statement
While Large Language Models (LLMs) have demonstrated high performance on standardized medical benchmarks, their reliability in complex, real-world oncology treatment planning remains uncertain. Breast cancer treatment requires synthesizing heterogeneous data (guidelines, pathology, patient history) to make nuanced decisions regarding systemic therapy, surgery, radiotherapy, and complementary exams. Previous studies indicate that LLMs often struggle with treatment sequencing, integrating surgical history, and achieving concordance with multidisciplinary tumor boards. Furthermore, the efficacy of "agentic" approaches—systems utilizing external tools (e.g., web/PubMed search) and multi-agent architectures (e.g., fact-checking, autonomous subagents)—in improving clinical recommendation quality is underexplored. A significant barrier to rigorous evaluation in this domain is the high cost and scarcity of expert-authored evaluation rubrics.

Methodology
The study evaluated agentic LLM systems using a dataset of 72 real clinical cases of invasive breast cancer (stages I–IV) from a private oncology clinic in Brazil. The evaluation framework consisted of three core components:

1. Asymmetric Information Rubric Generation (AIRG): To address the scarcity of expert rubrics, the authors introduced AIRG. In this workflow, a frontier LLM (gemini-3.5-flash) with high reasoning effort and access to privileged information (the actual clinical decisions made by oncologists and access to search tools) generated case-specific evaluation rubrics. The evaluated models received only the case summary, creating an information asymmetry designed to ground the rubrics in reference decisions without requiring manual expert creation for every case. This process yielded 1,147 case-specific rubrics across four categories: systemic therapy, radiotherapy, surgical therapy, and complementary exams.
2. Pipeline Architectures: Seven distinct pipelines were compared:
   • Baselines: Single-LLM prompting without tools.
   • Tool-Augmented: Single LLMs with access to Web Search, PubMed Search, or PubMed+PMC full-text retrieval.
   • Multi-Agent Architectures:
     • Divide and Conquer (D&C): An orchestrator agent delegating tasks to four fixed subagents (one per clinical domain), all with tool access.
     • D&C + Fact Checker (D&C+FC): Adds a fact-checking agent to verify claims before final output.
     • D&C + Auto-Spawning (D&C+SA): Agents autonomously spawn subagents with customized instructions, followed by a fact-checking step.
3. Models and Metrics: The study tested models from Google (gemini-2.5-flash, gemini-3.1-pro, gemini-3.5-flash), OpenAI (gpt-5.5-2026-04-23), and Anthropic (claude-opus-4-8). Performance was measured using a rubric-based scoring system (inspired by HealthBench), where a grader model assigns scores based on whether specific criteria are met, normalized to a 0–1 scale.

Key Results

• Overall Performance: The best-performing configuration was Claude Opus 4.8 using the D&C+SA pipeline, achieving a global score of 0.594 ± 0.025. The gap between the top frontier models was small.
• Impact of Agentic Features: The addition of tools and increased agent autonomy yielded mixed results. While D&C+SA performed best with Claude, other models (e.g., gemini-2.5-flash) saw performance degradation when tools were introduced. This suggests that agentic architectures are not universally beneficial and depend heavily on the base model's capability to manage tool interactions and reasoning traces.
• Domain and Stage Variance: Performance varied significantly by clinical domain and disease stage.
  • Domains: Scores were highest for systemic therapy (39% of rubrics) and lowest/most variable for surgery and radiotherapy.
  • Stages: Stage I cases yielded the highest mean scores (0.545), while Stage III cases yielded the lowest (0.441), likely reflecting the increased complexity and diversity of treatment options in advanced stages.
• Error Analysis: A qualitative review by a certified oncologist of 10 top-performing responses revealed 8 recurrent failure modes, including:
  • Incorrect or incomplete justifications (22.7%).
  • Incorrect or missing clinical recommendations (20.0%).
  • Citation errors and reference fabrication (16.0% and 2.7% respectively).
  • Overconfident statements and outdated claims.
  • The review required approximately 1.5 hours per case, highlighting the difficulty of manual verification.

Key Contributions

1. Benchmarking Agentic Systems: The study provides a comprehensive comparison of single-LLM, tool-augmented, and multi-agent pipelines for breast cancer treatment, demonstrating that architectural complexity does not linearly correlate with performance.
2. AIRG Workflow: The introduction of Asymmetric Information Rubric Generation offers a pragmatic solution for generating evaluation criteria in specialized domains where expert time is a bottleneck, leveraging reference decisions to ground the rubrics.
3. Error Characterization: The paper identifies specific, clinically relevant failure modes (e.g., citation hallucinations, overconfidence) that persist even in top-performing systems, emphasizing that aggregate scores can mask critical safety risks.
4. Open Source: The authors released source code to support reproducibility and future work in clinically grounded AI evaluation.

Significance and Claims
The paper concludes that while agentic LLM systems can generate clinically relevant breast cancer recommendations, they remain insufficient for unsupervised clinical use. The findings suggest that:

• Model Capability is Paramount: Base model strength is a primary determinant of success; architectural improvements (tools, agents) cannot reliably compensate for weaker models.
• Context-Dependent Design: Agentic architectures should not be treated as universal solutions; their efficacy is model-dependent and task-dependent.
• Evaluation Necessity: Aggregate benchmark scores are inadequate for clinical deployment. Stratified evaluation (by stage and domain) and qualitative error analysis are essential to uncover hidden weaknesses.
• Safety Risks: Persistent issues such as citation fabrication, overconfidence, and outdated claims pose significant risks, necessitating human oversight and hybrid evaluation workflows before any clinical integration.

The authors maintain a modest stance, acknowledging limitations such as the single-clinic dataset, the reliance on automatically generated rubrics, and the lack of adversarial testing, while calling for future work to validate AIRG against expert rubrics and develop more robust, scalable evaluation frameworks.