Colon-X: Advancing Intelligent Colonoscopy toward Clinical Reasoning

1. Problem Statement

While intelligent colonoscopy has advanced in low-level vision tasks (e.g., polyp segmentation), current Multimodal Large Language Models (MLLMs) struggle to transition from simple multimodal understanding to clinically grounded reasoning. The paper identifies three critical gaps:

• Data Scarcity & Fragmentation: Existing datasets are small, lack task diversity, and do not support complex reasoning traces.
• Unreliable Generalizability: It is unclear how well existing MLLMs (both open and closed-source) perform across diverse clinical tasks, particularly regarding robustness against human-induced biases.
• Reasoning Instability: Applying standard reinforcement learning (like GRPO) to medical reasoning tasks often leads to optimization collapse due to "reward information collapse," where the model fails to distinguish between heterogeneous task structures or difficult cases.

2. Methodology

The COLON-X project proposes a unified framework addressing data, evaluation, and model optimization through four core components:

A. Data Foundation: COLONVQA

The authors constructed COLONVQA, the largest multimodal colonoscopy dataset to date.

• Scale: 212,742 images and 1.1 million+ VQA entries (49.9M+ tokens).
• Taxonomy: Organized into a 5-level taxonomy covering 18 clinical tasks and 76 categories.
  • Quality Control: Bowel cleanliness, anatomical landmarks.
  • Safety Monitoring: Instrument recognition, bleeding alerts.
  • Lesion Diagnosis: Detection, classification, and open-ended description.
  • Disease Grading: Cancer staging (NICE/PARIS criteria), polyp sizing, ulcerative colitis scoring.
  • Documentation: Image captioning.
• Preprocessing: Includes rigorous label standardization, redundancy removal, and strict patient-level data splitting to prevent leakage.

B. Evaluation: Generalizability & Reliability

Two benchmarks were introduced to assess current MLLM capabilities:

1. COLONEVAL (Generalizability): A subset of 4,568 VQA entries used to evaluate 22 MLLMs (13 open-source, 9 closed-source).
   • Findings: Closed-source models generally outperform open-source ones, but open-source models excel in specific safety tasks. Interestingly, "generalist" models sometimes outperform "medical specialist" models, and reasoning-enabled variants do not always guarantee higher accuracy.
2. COLONPERT (Reliability): A perturbation test suite to measure robustness against text-dominance bias.
   • Implicit Bias: Masking or replacing text embedded in images (e.g., device metadata).
   • Explicit Bias: Injecting case-contradicting instructions or emotionally charged narratives (e.g., patient anxiety) into prompts.
   • Findings: MLLMs are highly susceptible to text-based biases, often ignoring visual evidence when textual prompts are misleading or emotionally manipulative.

C. Reasoning Data: COLONREASON

To overcome the cost of human-annotated reasoning traces, the authors developed a Multi-Agent Debating Pipeline:

• Process: A 5-step loop involving (1) Initial interpretation by two agents, (2) Peer debating/critique, (3) Self-reflection and update, (4) Aggregation of consensus, and (5) Human adjudication.
• Output: Generates structured reasoning traces (Chain-of-Thought) verified by clinicians for correctness, visual grounding, and auditability.
• Scale: ~7,500 high-quality reasoning samples derived from COLONVQA.

D. Model: COLONR1

The authors propose COLONR1, the first R1-style model tailored for colonoscopy reasoning, designed to solve optimization instability in medical tasks.

• Task-Adaptive Rewarding: Replaces binary rewards with a composite scheme (continuous scores for open questions, graded scores for multiple-choice) to maintain discriminative signals across diverse tasks.
• Negative Sampling: Actively injects incorrect answers into the training group to prevent "advantage vanishing" in easy cases where all samples might be correct.
• Self-Evolving Memory: A mechanism that stores hard cases (low reward) and evolves the prompt for future rollouts, forcing the model to learn from past failures.
• Training: Reinforced Fine-Tuning (GRPO) on a Qwen2.5-VL-3B base model using only ~7.5K samples.

3. Key Results

• Dataset Scale: COLONVQA is the most extensive colonoscopy dataset, surpassing previous benchmarks (e.g., Kvasir-VQA, EndoVQA) in image count, task diversity, and VQA volume.
• Evaluation Insights:
  • Closed-source models (e.g., Gemini 2.5 Flash) achieved the highest overall accuracy (~73%), while open-source models showed surprising strength in safety monitoring.
  • Models suffered significant accuracy drops (up to 90% for some open models) when exposed to on-image text masking or misleading instructions, highlighting a lack of robustness.
• Model Performance:
  • COLONR1 achieved 56.61% overall accuracy on the COLONEVAL benchmark.
  • This represents a 25.22% improvement over standard Supervised Fine-Tuning (SFT) and a significant gain over naive GRPO approaches.
  • It outperformed the best baseline (Med-R1) by a large margin, demonstrating the effectiveness of the task-adaptive rewards and memory mechanisms.

4. Significance & Contributions

• Paradigm Shift: Moves the field from "Multimodal Understanding" (classification/detection) to "Clinical Reasoning" (interpretation and decision support).
• Open Initiative: Provides a comprehensive, open-source ecosystem (Data, Evaluation, Model) to accelerate research in medical AI.
• Methodological Innovation:
  • Demonstrates that multi-agent debating can effectively synthesize high-quality reasoning data without massive human annotation costs.
  • Identifies and solves reward information collapse in medical reinforcement learning, offering a stable training framework for complex, heterogeneous clinical tasks.
• Clinical Safety: Highlights the critical vulnerability of current AI to "text-dominance bias," urging future systems to prioritize visual grounding and robustness against emotional or misleading prompts.

Conclusion

COLON-X establishes a new benchmark for intelligent colonoscopy. By combining a massive, diverse dataset with a robust evaluation suite and a specialized reasoning model, the authors provide a roadmap for developing trustworthy, clinically grounded AI assistants that can support, rather than just mimic, expert medical decision-making.