Evaluating the Large Language Model-Based Quality Assurance Tool for Auto-Contouring

This study demonstrates that the LAQUA system, a multimodal large language model-based tool, achieves substantial agreement with expert radiation oncologists in evaluating auto-contouring quality, suggesting its feasibility as an efficient primary screening tool to reduce clinical workload despite a potential risk of overestimating contour acceptability.

The Gist

Imagine you are a master chef preparing a massive banquet for 20 different guests. You have hired three different sous-chefs (AI software) to chop the vegetables and plate the food. They are incredibly fast and usually do a great job. However, sometimes they get distracted, slice a carrot too thin, or forget to peel a potato.

If you, the head chef, have to check every single plate for every single guest, it takes hours. You might get tired, your eyes might glaze over, and you could accidentally serve a plate with a raw potato because you missed it. This is exactly the problem doctors face with AI auto-contouring in radiation therapy.

The Problem: The Tired Eye

In radiation therapy, doctors must draw precise outlines (contours) on CT scans to tell the radiation machine exactly where to aim and where to avoid. AI can draw these outlines in seconds, but it's not perfect. Sometimes the AI gets confused by weird anatomy or image glitches.

Traditionally, a human doctor has to look at hundreds of slices of a CT scan to verify the AI's work. It's tedious, boring, and prone to human error (like missing a mistake because you're tired).

The Solution: The "AI Critic" (LAQUA)

The researchers in this paper built a new tool called LAQUA. Think of LAQUA as a super-smart, tireless food critic who is also a master chef.

Instead of just checking if the food looks "mathematically correct" (like measuring the exact millimeters of a carrot slice), LAQUA uses a Large Language Model (LLM)—a type of AI that is very good at understanding language and images.

Here is how it works:

1. The Setup: The AI sous-chefs (OncoStudio, RatoGuide, and syngo.via) draw their outlines on the CT scans.
2. The Inspection: LAQUA looks at these drawings. It doesn't just calculate numbers; it "sees" the image and "reads" the context.
3. The Verdict: LAQUA gives each drawing a grade from 1 to 5:
   • 5: Perfect! Serve it immediately.
   • 4: Good, just a tiny tweak needed.
   • 3: Okay, but needs some work.
   • 2: Bad. Throw it out and start over.
   • 1: Disaster. The AI didn't even see the organ.
4. The Explanation: This is the magic part. If LAQUA gives a low score, it doesn't just say "Error." It writes a note in plain English, like: "The outline of the prostate is too high up near the top," or "The rectum is missing a piece because of gas in the intestine."

What Did They Find?

The researchers tested this system on 20 male pelvic cases (think of it as a tasting menu of 20 different meals).

• It Agrees with the Experts: When two real human doctors graded the same drawings, LAQUA's grades matched theirs very closely. It was about 75-80% in sync with human experts.
• It's a Great "Gatekeeper": The goal wasn't to replace the human doctor, but to act as a security guard at the door. LAQUA is really good at spotting the "bad" plates. If LAQUA says, "This one looks fine," the human doctor can trust it enough to do a quick check rather than a deep dive. If LAQUA says, "This one is broken," the human doctor knows exactly where to look.
• It Can Hallucinate: Like any AI, LAQUA isn't perfect. Sometimes it gets confused by things like gas bubbles in the body and invents a problem that isn't there (a "hallucination"). However, this happened rarely.

The Big Picture

Think of LAQUA as a spell-checker for radiation therapy.

Before, if you wrote a 100-page document, you had to read every word to find typos. Now, you can run it through a spell-checker. The spell-checker might miss a few subtle errors, but it will catch the obvious typos and tell you, "Hey, you spelled 'radiation' wrong on page 42."

This allows the human doctor to focus their energy on the tricky parts, rather than getting bored checking the easy parts.

The Conclusion

The study concludes that this "AI Critic" is a promising tool. It won't replace the human doctor (the head chef), but it will save them hours of work and make the final meal (the treatment plan) safer for the patient by ensuring no "raw potatoes" slip through the cracks.

In short: We taught a super-smart AI to look at medical drawings, grade them, and explain why they are good or bad, helping human doctors work faster and safer.

Technical Summary

1. Problem Statement

The integration of Artificial Intelligence (AI) for auto-contouring (AC) in radiotherapy has significantly reduced manual labor and inter-operator variability. However, AC systems are not infallible; they can produce inaccurate contours due to image artifacts, anatomical variations, or training data biases.

• Current Bottleneck: Manual verification by radiation oncologists is labor-intensive, prone to fatigue, and susceptible to "automation bias" (where experts overlook errors because they trust the AI).
• Limitations of Existing AI QA: Previous attempts to automate QA using geometric metrics (e.g., Dice Similarity Coefficient) often fail to correlate with clinical significance. Other methods that output simple "pass/fail" judgments or use fixed templates lack the ability to provide context-specific, natural language explanations of why a contour is flawed.

2. Methodology

The study developed and evaluated LAQUA (LLM-based Automated Quality Assurance for Auto-Contouring), a system utilizing a multimodal Large Language Model (Gemini 2.5 Pro) to assess contour quality.

• Dataset:
  • 20 male pelvic CT scans from a public, de-identified open dataset.
  • Target structures: Bladder, prostate, rectum, and bilateral femoral heads.
  • The dataset included "edge cases" to test robustness.
• Auto-Contour Generation:
  • Contours were generated using three distinct commercial software packages: OncoStudio, RatoGuide (prototype), and syngo.via.
  • Default settings were used for all platforms.
• System Architecture (LAQUA):
  • Input Processing: Generated contours were overlaid as red lines on CT slices (Window Level: 40 HU, Width: 350 HU) and converted into PDFs. To preserve 3D context, full fields of view were kept, and three additional slices beyond the organ boundaries were included.
  • LLM Inference: The PDFs were fed into Gemini 2.5 Pro via API.
  • Prompt Engineering: The LLM was instructed to act as a radiation therapy expert, scoring each structure on a 5-point clinical scale (5: Optimal to 1: Organ not detected) and providing a natural language rationale for the score.
  • Parameters: Temperature set to 0.1 to balance reproducibility and stability.
• Evaluation Metrics:
  • Geometric Baseline: Calculated Volume Dice (vDSC), 95th percentile Hausdorff Distance (HD95), and Surface Dice (sDSC) against ground truth.
  • Quantitative Agreement: Compared LLM scores against consensus scores from two board-certified radiation oncologists using Spearman's rank correlation ($\rho$) and quadratic weighted Kappa ($\kappa$).
  • Screening Performance: Calculated sensitivity and specificity by dichotomizing scores (defining "adequate" as $\ge3$ or $\ge4$).
  • Qualitative Analysis: Experts rated the LLM's rationales on a 2-point Likert scale across four domains: error detection, hallucination, clinical relevance, and anatomical understanding.

3. Key Contributions

• Novel Workflow: First study to implement a fully automated, multimodal LLM pipeline for clinical auto-contouring QA that outputs structured scores and natural language rationales rather than just geometric metrics or binary pass/fail flags.
• Human-Interpretable Feedback: The system provides specific, context-aware descriptions of errors (e.g., "cranial boundary overestimated"), potentially guiding human reviewers more effectively than raw numbers.
• Comprehensive Validation: Validated the system across three different commercial AC software packages and five anatomical structures, including a qualitative assessment of the LLM's reasoning capabilities.

4. Results

• Geometric Baseline: All three software platforms showed good geometric agreement (mean vDSC $\ge$ 0.8), though outliers existed, particularly in prostate delineation.
• Agreement with Experts:
  • The LAQUA system demonstrated strong agreement with human experts.
  • Correlation: Spearman's $\rho$ ranged from 0.733 to 0.794 across software; Quadratic weighted $\kappa$ ranged from 0.730 to 0.798.
  • Organ Specificity: Highest agreement was found in the rectum ($\rho=0.835$), while the left femoral head showed the lowest ($\rho=0.567$).
• Screening Performance:
  • When defining "adequate" as a score $\ge4$ (requiring minor modification), the system achieved high sensitivity in the rectum (0.976) and high specificity in the left femoral head (0.933).
  • Confidence intervals narrowed when using the stricter $\ge4$ cutoff, suggesting better reliability for filtering "acceptable but modifiable" cases.
• Qualitative Rationale Evaluation:
  • The LLM's rationales received a mean score of 1.70/2.0.
  • 155 out of 291 outputs received perfect scores (2/2) across all criteria.
  • Failure Modes: The system occasionally hallucinated errors (e.g., misinterpreting gas as a contour error) or missed context due to a lack of specialized radiotherapy atlas knowledge.

5. Significance and Conclusion

• Clinical Utility: The LAQUA system is not intended to replace human experts but to serve as a primary screening tool. It can efficiently filter out "acceptable" contours, allowing human reviewers to focus only on cases flagged as "suboptimal" or "unacceptable."
• Mitigating Automation Bias: By explicitly verbalizing error locations and reasons, the system helps break the "automation bias" where humans might overlook AI errors, thereby reducing the risk of clinical incidents.
• Future Directions: The authors suggest that integrating Retrieval-Augmented Generation (RAG) with specific radiotherapy contouring guidelines could mitigate hallucinations and improve performance in specialized domains.
• Limitations: The study was limited to a small, public male pelvic dataset. The conversion of 3D DICOM data to 2D PDFs for LLM input may have resulted in the loss of inter-slice continuity and contrast information.

Conclusion: The study demonstrates that multimodal LLMs can effectively evaluate auto-contouring quality with substantial agreement to human experts, offering a promising pathway to streamline the QA workflow in radiotherapy.