From Clinical Intent to Clinical Model: An Autonomous Coding-Agent Framework for Clinician-driven AI Development

1. Problem Statement

Traditional clinical AI development relies on a collaborative paradigm where clinicians define the problem and AI engineers translate it into technical implementation. This workflow suffers from significant bottlenecks:

• Communication Overhead: Iterative discussions are time-consuming to refine requirements.
• Misalignment: Clinicians often lack data science expertise, while AI developers may miss critical clinical priorities (e.g., specific failure modes or risk preferences).
• Shortcut Learning: Models may optimize for aggregate metrics while relying on spurious correlations (e.g., detecting chest drains instead of pneumothorax signs) because the clinical context isn't fully encoded in the technical objective.

The paper proposes shifting this paradigm to a clinician-driven model where clinicians can specify tasks and constraints in natural language, and an autonomous agent handles the entire development lifecycle.

2. Methodology: The Autonomous Coding-Agent Framework

The authors present a prototype system that converts natural language clinician requests into trained, task-specific AI models through three sequential stages:

A. System Architecture

1. Semantic Parser: Converts the clinician's unstructured natural language request (e.g., "Do not miss melanomas," "Avoid relying on chest drains") into a structured task representation.
2. Task Initializer: Translates the structured representation into an executable codebase, defining the model architecture, training recipe, and evaluation protocol.
3. Autonomous Developer: An iterative loop where the agent:
   • Executes the current codebase.
   • Analyzes validation results.
   • Proposes code modifications (debugging, hyperparameter tuning, architecture changes, loss function adjustments).
   • Accepts changes only if they improve the validation metric.
   • Constraint: The agent operates within a fixed budget (e.g., 30 iterations, 40-minute run time per iteration) and never sees the held-out test set.

B. Implementation Details

• Base Model: The system utilizes Claude Opus 4.6 as the underlying Large Language Model (LLM) agent, performing all three roles (parser, initializer, developer) via prompt engineering.
• Hardware: Experiments were conducted on two NVIDIA L40S GPUs.
• Evaluation Protocol: A strict "hold-out" protocol was used. The test set was invisible to the agent throughout the 30-round refinement process. Metrics were calculated only once after the final model was selected.

3. Key Contributions

• Proof of Concept for Clinician-Driven AI: Demonstrates that non-technical clinicians can drive the development of deep learning models using only natural language, bypassing the need for a specialized AI team.
• Autonomous Handling of Complex Constraints: The system successfully interpreted and acted upon high-level clinical constraints, such as prioritizing sensitivity over specificity or actively mitigating known confounders (shortcut learning).
• Weak Supervision Adaptation: The agent autonomously recognized a mixed-supervision scenario (5% bounding boxes, 95% image-level labels) and constructed a teacher-student pseudo-labeling strategy without explicit instruction.
• Debiasing Implementation: The agent successfully implemented a multi-component debiasing strategy (gradient reversal, group-balanced sampling, mixup) to reduce reliance on spurious correlations (chest drains) based solely on a clinician's warning.

4. Results

The framework was evaluated on five clinical tasks across three categories:

A. Well-Defined Supervised Tasks

1. 8-Class Dermoscopic Lesion Classification:
   • Request: Prioritize melanoma.
   • Outcome: The agent introduced Focal Loss to emphasize melanoma.
   • Performance: Melanoma AUC increased from 0.842 to 0.916; Overall AUC from 0.879 to 0.915.
2. Melanoma vs. Nevus Triage:
   • Request: "Do not miss a single case" (High sensitivity).
   • Outcome: The agent optimized for sensitivity at fixed specificity.
   • Performance: Sensitivity at 80% specificity improved from 0.602 to 0.909.
3. Wrist Fracture Detection:
   • Request: Detect fractures and mark them on images.
   • Outcome: Improved localization quality.
   • Performance: mAP@50:95 increased from 0.471 to 0.511.

B. Mixed-Supervision Task (Weak Labels)

• Wrist Fracture (5% Bounding Boxes):
  • Challenge: Only 5% of training data had bounding boxes; 95% had only image-level labels.
  • Agent Action: The agent autonomously constructed a teacher-student pseudo-labeling strategy, utilizing the 95% weakly labeled data.
  • Outcome: mAP@50 improved from 0.794 to 0.852; Recall increased significantly from 0.507 to 0.717, aligning with the clinical goal of minimizing missed fractures.

C. Shortcut Mitigation Task (Debiasing)

• Pneumothorax Classification (Chest Drains as Confounder):
  • Challenge: Chest drains are inserted after diagnosis, creating a spurious correlation. The clinician explicitly requested the model not rely on them.
  • Agent Action: The agent implemented a multi-step debiasing strategy:
    1. Group-balanced sampling to balance drain presence across disease states.
    2. Gradient Reversal Layer (Adversarial Training) to force the backbone to learn features predictive of pneumothorax but uninformative about drain presence.
    3. Mixup and Label Smoothing to reduce overconfidence.
  • Outcome:
    • False positives attributable to chest drains dropped from 60% to 31%.
    • Partial correlation between model output and drain presence (controlling for true disease) decreased by 47% (from 0.48 to 0.26).

5. Significance and Limitations

Significance

• Paradigm Shift: Moves clinical AI development from a "black box" translation process to a transparent, intent-driven workflow.
• Accessibility: Lowers the barrier to entry for domain experts (clinicians) who understand the clinical problem but lack deep learning implementation skills.
• Safety & Alignment: Demonstrates that autonomous agents can actively mitigate safety risks (shortcut learning) when explicitly instructed, potentially reducing deployment failures.

Limitations & Future Work

• Reliance on Existing Knowledge: The agent acts as an "integrator" of known techniques (e.g., Focal Loss, Gradient Reversal) rather than a discoverer of novel algorithms. It struggles if the solution requires undocumented institutional knowledge or highly specialized, non-standard methods.
• Semantic Parsing Fidelity: The system's success depends entirely on the accuracy of translating vague clinical intent into precise mathematical objectives.
• Generalization: Current validation is limited to public benchmarks. Real-world deployment requires testing on heterogeneous, multi-center data where "shortcuts" may be more diffuse and harder to identify.
• Human-in-the-Loop: The authors argue the agent should not replace specialist oversight but rather serve as an interface to accelerate iteration and reduce translation burdens.

Conclusion

The study provides a robust proof of concept that autonomous coding agents can translate high-level clinical intents into functional, optimized deep learning models. By successfully handling complex constraints like weak supervision and confounder mitigation, the framework suggests a viable path toward more efficient, clinician-centric, and safer AI development in medicine.