IOSVLM: A 3D Vision-Language Model for Unified Dental Diagnosis from Intraoral Scans

This paper introduces IOSVLM, an end-to-end 3D vision-language model that leverages native point cloud geometry and a specialized training strategy to achieve unified multi-disease diagnosis and visual question answering on intraoral scans, supported by the newly proposed large-scale IOSVQA dataset.

The Gist

Imagine you are a detective trying to solve a mystery inside a patient's mouth. Traditionally, dentists have used 2D photos (like flat snapshots) or physical models to find problems. But today, we have Intraoral Scans (IOS): these are like high-definition, 3D holographic maps of a person's teeth and gums. They show every tiny curve, crack, and gap in 3D space.

The problem? Most current "AI detectives" are trained to look at flat 2D photos. When you force them to look at a 3D hologram, they have to squint and guess, often missing the subtle details that only exist in the 3D shape.

IOSVLM is a new, super-smart AI detective designed specifically to read these 3D holographic maps directly. Here is how it works, explained simply:

1. The Problem: The "Color Blindness" of 3D Scans

Most AI models that understand 3D shapes were trained on colorful objects (like a red apple or a blue car). They expect to see colors to help them distinguish edges.

• The Issue: Dental scans are often just "white" 3D shapes. They don't have reliable colors. If you feed a color-hungry AI a white 3D tooth, it gets confused, like a painter trying to paint a masterpiece with only white paint.
• The Solution (The "Magic Paintbrush"): The researchers invented a trick called the Geometry-to-Chromatic Proxy. Instead of real colors, they use the shape of the tooth to create "fake colors."
  • Analogy: Imagine running your finger over a bumpy rock. Even if the rock is all one color, your finger feels the bumps and valleys. The AI does the same thing: it uses the "bumps" (curvature and angles) to create a visual map that tells the AI, "Hey, this is a sharp edge!" or "This is a smooth curve!" This lets the AI use its existing 3D training even without real colors.

2. The Challenge: The "Messy Room" of Dental Data

Dental scans are messy. Sometimes the scanner only sees the top teeth (Single-Arch), and sometimes it sees the top and bottom teeth smashed together (Occluded-Arch). Plus, one patient might have five different problems at once (cavities, crooked teeth, gum disease).

• The Analogy: Imagine trying to learn a language by reading a book where some pages are missing, some are written in different fonts, and some sentences have typos.
• The Fix (The "Two-Stage School"): The AI doesn't try to learn everything at once. It uses a Curriculum Learning strategy:
  • Stage 1 (The Boot Camp): The AI studies a huge pile of data, even if it's a bit messy or noisy. It learns the basic "grammar" of 3D shapes and how to talk about them.
  • Stage 2 (The Advanced Class): The AI then studies a smaller, high-quality set of data where the answers are perfect. It refines its skills, learning to be precise and to explain why it made a diagnosis (like a doctor writing a report).

3. The Result: A New Super-Doctor

The researchers built a massive dataset called IOSVQA with nearly 20,000 cases and over 249,000 questions and answers. They trained IOSVLM on this.

How did it do?

• The Competition: They tested IOSVLM against other famous AI models (like GPT-5 and Gemini).
• The Outcome: IOSVLM crushed the competition.
  • While other models tried to flatten the 3D scan into 2D pictures (like taking a photo of a sculpture from the side), IOSVLM looked at the sculpture itself.
  • It was 15% more accurate than the best open-source models and even beat some of the massive, expensive "closed" AI models, despite being smaller.
  • It didn't just say "Yes, there is a problem"; it could say, "Yes, there is a gap between the front teeth, and here is why," mimicking a real dentist's report.

Why This Matters

Think of IOSVLM as the difference between looking at a flat map of a city versus walking through the city in 3D.

• Old AI: Looks at a 2D map, guesses where the potholes are, and often misses them.
• IOSVLM: Walks through the 3D city, feels the bumps, sees the cracks in the pavement, and gives you a detailed, accurate report.

This technology means that in the future, dentists could scan a patient's mouth, and an AI could instantly generate a full, accurate diagnosis and a clear explanation for the patient, catching subtle problems that human eyes might miss. It's not just about finding disease; it's about understanding the complex, 3D reality of our mouths.

Technical Summary

1. Problem Statement

The paper addresses the challenge of unified multi-disease diagnosis using 3D Intraoral Scans (IOS). While IOS provides high-fidelity 3D geometric data superior to 2D imaging, existing solutions face significant limitations:

• 2D Limitations: Current Vision-Language Models (VLMs) rely on 2D images or multi-view renderings of 3D scans, which discard native geometric cues and spatial relationships.
• 3D Challenges: Directly modeling native 3D IOS is difficult due to:
  1. Heterogeneity: Scans vary significantly (single-arch vs. occluded-arch) with complex topology and occlusion.
  2. Complexity: Multiple diseases often co-occur in a single scan with class imbalance and fine-grained morphological ambiguity.
  3. Data Scarcity: There is a severe lack of paired 3D IOS–text data for training.
  4. Pretraining Mismatch: Most 3D point-cloud models are pretrained on colored data (RGB), but clinical IOS data often lacks reliable color/texture, causing distribution shifts.

2. Methodology

A. Dataset: IOSVQA

The authors constructed IOSVQA, the first large-scale, multi-source diagnostic VQA dataset for IOS.

• Scale: 19,002 cases and 249,055 VQA pairs.
• Scope: Covers 23 oral diseases and two scan types (single-arch and occluded-arch).
• Sources: Aggregated from private clinical data (MaloccIOS, DiseaseIOS) and public data (Bits2Bites).
• Annotation: Includes high-quality labels and Chain-of-Thought (CoT) rationales generated by LLMs and verified by experts for a subset of data to enhance reasoning.

B. Model Architecture: IOSVLM

IOSVLM is an end-to-end 3D VLM following an Encoder-Projector-LLM design:

1. 3D Encoder: Uses ReCON++ (a pre-trained 3D point-cloud encoder) to extract multi-scale features:
   • Absolute position embeddings ($F_{ape}$).
   • Local geometric features ($F_{local}$).
   • Global descriptors ($F_{global}$).
2. Projectors: Three dedicated MLPs map these features into the LLM token space. Learnable visual prompts are concatenated to balance contributions.
3. LLM: A large language model (based on Qwen3VL-8B) performs reasoning and generates diagnostic outputs (labels and rationales).

C. Key Technical Innovations

• Geometry-to-Chromatic Proxy (GCP):
  • Problem: Standard 3D encoders expect RGB inputs, but IOS data is often colorless. Naively padding with zeros causes distribution shifts.
  • Solution: GCP replaces RGB channels with surface normals (geometry-derived). Normals are normalized and flipped to create a "pseudo-color" map that mimics the separability cues of RGB without semantic color. This allows the model to reuse pretraining priors effectively.
• Two-Stage Curriculum Training:
  • Stage 1 (Alignment): Trains the 3D encoder and projectors on large-scale, noisy data to build robust geometric representations and geometry-language alignment. The LLM is frozen.
  • Stage 2 (Instruction Tuning): Fine-tunes the projectors and LLM (using LoRA) on high-quality data, including CoT rationales, to enhance diagnostic reliability and interpretability.

3. Key Contributions

1. First Large-Scale IOS VQA Dataset: Created IOSVQA, covering 23 diseases and heterogeneous scan types, explicitly modeling real-world co-existing disease scenarios.
2. End-to-End 3D VLM: Introduced IOSVLM, the first model to take native 3D IOS geometry as input for unified diagnosis, outperforming 2D rendering-based approaches.
3. Geometry-to-Chromatic Proxy: Proposed a novel method to bridge the gap between color-free clinical scans and color-dependent 3D pretraining, significantly improving fine-grained geometric perception.

4. Experimental Results

The model was evaluated on IOSVQA against strong baselines, including proprietary MLLMs (GPT-5, Gemini 3 Pro), open-source 2D MLLMs, and open-source 3D MLLMs.

• Performance: IOSVLM achieved state-of-the-art results:
  • Macro Accuracy: 77.23% (vs. 67.65% for Gemini 3 Pro; +9.58% gain).
  • Macro F1: 50.39% (vs. 48.93% for Gemini 3 Pro; +1.46% gain).
  • Recall: 52.96% (highest among all baselines).
• Comparison:
  • Outperformed open-source 2D MLLMs by a large margin (>16% Acc).
  • Surpassed open-source 3D MLLMs by >34% Acc, proving the efficacy of the specific architecture and GCP.
  • Achieved 100% Parsing Rate (PR), meaning all generated outputs were valid labels, whereas proprietary models occasionally produced unparseable empty outputs.
• Ablation Studies:
  • GCP: Adding the proxy improved Accuracy by +5.26% and F1 by +4.96%, confirming that geometric separability cues are critical.
  • Training Strategy: The two-stage curriculum proved superior to single-stage training, balancing geometric learning with high-quality reasoning.

5. Significance

• Clinical Utility: IOSVLM enables unified, multi-disease diagnosis directly from 3D scans, reducing the need for manual segmentation or single-lesion detection. It generates natural language reports suitable for dentist-patient communication.
• Geometric Modeling: The paper demonstrates that modeling native 3D geometry is superior to 2D rendering for dental tasks, as it preserves fine-grained morphological details essential for detecting subtle abnormalities.
• Data Efficiency: The GCP technique allows the effective transfer of knowledge from color-pretrained 3D models to colorless clinical data, a crucial step for deploying 3D AI in real-world medical settings where texture data is often discarded.
• Benchmarking: The release of IOSVQA provides a standardized benchmark for future research in 3D dental AI, addressing the previous lack of large-scale paired 3D-text data.