Million-scale multimodal pollen microscopy with expert-guided foundation models

Technical Summary: Million-scale Multimodal Pollen Microscopy with Expert-Guided Foundation Models

Problem Statement
Automated pollen identification via microscopy remains a critical bottleneck in aerobiology, palaeoecology, and biodiversity monitoring. Existing scalable systems struggle to generalize across variations in specimen preparation, scanner settings, and geographic origins while maintaining palynological interpretability. Current public resources are often constrained by limited taxonomic breadth, sample counts, or imaging modalities, and most lack structured, natural-language morphological records paired at the grain level. Furthermore, traditional supervised detector training is brittle when deployed on taxa or conditions differing from the labeled training regime. There is a lack of a large-scale, acquisition-diverse, expert-grounded resource that pairs individual bright-field pollen crops with structured natural-language descriptions.

Methodology
The authors present Pollen AI Atlas, a million-scale multimodal resource constructed from pure-species whole-slide bright-field images. The workflow relies on "weak supervision" using pure-species slides, where slide-level taxonomic labels serve as context without requiring exhaustive per-grain annotation. The pipeline consists of five stages:

1. One-Shot Initialization: For each source slide, a single manually selected pollen grain exemplar seeds the process. This exemplar is used with OWL-ViT to generate initial candidate bounding boxes via image-guided detection.
2. Token-Level Mining: Instead of training a task-specific detector, the system mines dense visual representations directly. A manually selected exemplar seeds a search in the feature space of a fine-tuned DINOv2-based LVD-ViT-S backbone. The system constructs a query representation in token space, matches it against whole-slide tokens using cosine similarity, and iteratively extracts candidate objects.
3. Quality Filtering: A four-stage filtering pipeline removes debris and duplicates:
   • Attention-based re-embedding: Candidates are re-embedded via cross-attention pooling to create uniform representations.
   • Prototype-based reranking: Iterative medoid computation refines a slide-level prototype to reduce exemplar bias.
   • Non-maximum suppression (NMS): Spatial overlaps are removed with a low IoU threshold (0.10).
   • Classifier gating: A fine-tuned LVD-ViT-S classifier acts as a semantic gate to reject background and low-confidence detections.
4. Vision-Language Captioning: The filtered detections are processed by five open-weight Vision-Language Models (VLMs): Qwen2.5-VL, Qwen3-VL, Qwen3.5, Qwen3.6, and Gemma4. Generation is guided by expert-verified palynological anchors (derived primarily from Beug's Leitfaden der Pollenbestimmung and PalDat). Prompts explicitly prohibit taxon-name leakage and raw numeric measurements, enforcing structured descriptions of aperture systems, wall ornamentation, shape, and size.
5. Expert Ground Truth: A subset of regions (TS2) was manually curated by experts to evaluate detection precision and serve as a benchmark for downstream tasks.

Key Contributions

• Dataset Scale and Diversity: The release comprises 1,511,390 filtered pollen grain detections from 80 whole-slide images, spanning four geographic origins (France, Hungary, Mediterranean, Sweden), four scanner settings, and 46 taxon labels across 31 botanical families.
• Multimodal Alignment: Each detection is paired with machine-generated, expert-anchored morphological captions, creating the first million-scale image-caption microscopy resource for pollen.
• Weak-Supervision at Scale: The study demonstrates that a single manually selected exemplar per slide, combined with dense feature mining and filtering, can yield a high-precision corpus without exhaustive manual annotation.
• Benchmarking Suite: The release includes data, annotations, captions, splits, code, and weights, establishing benchmarks for pollen recognition, cross-regional domain adaptation, and domain-specific multimodal learning.

Results

• Detection Quality: The pipeline achieved 99.6% precision (20 false positives out of 4,506 true positives in expert-curated test regions) with a recall of 43.6%. Precision exceeded 99% in all regional subsets, confirming the "precision-first" operating point suitable for reference corpora.
• Caption Diagnostics: Among the five evaluated models, Gemma4 provided the most controlled output, with 0% taxon-name leakage, minimal numeric size leakage, and the strongest text-retrieval performance. It produced captions with a mean length of 70.6 words, tightly adhering to the 60–80 word constraint.
• Classification: A linear probe on frozen visual features achieved 88.16% Top-1 accuracy on the combined-origin test set. Single-origin training resulted in lower accuracy (66.16%–74.92%), highlighting the challenge of cross-regional generalization. Stain normalization (Macenko) improved performance for the Hungarian subset (+10.7%) but degraded the combined model (-21.2%), indicating that acquisition shifts are not uniform.
• Cross-Regional Retrieval: This was the most significant multimodal finding. In a "leave-origin-out" test (CROSS-REG), image-only retrieval performance dropped significantly (mAP@20 from 0.917 to 0.262). In contrast, text-based retrieval using Gemma4 captions remained robust (mAP@20 0.811), demonstrating that morphology-focused text embeddings are more invariant to scanner and preparation shifts than raw image features.

Significance and Claims
The paper claims that Pollen AI Atlas serves as a high-precision, expert-guided, pure-species reference resource for training and benchmarking, rather than an exhaustive detector for arbitrary mixed environmental samples. The authors assert that the atlas demonstrates how limited expert input (exemplar selection, anchor curation, sampled validation) can be combined with foundation models to create scalable, reusable resources for multimodal microscopy AI.

The work highlights that morphology-focused text is more robust across domain shifts than image similarity alone, mirroring expert palynological practice where identification relies on morphological features rather than acquisition-specific cues. The authors position the atlas not as a final solution for zero-shot operational detection, but as essential infrastructure for future pollen-adapted vision–language models and expert-guided analysis. The study extends previous detector-focused workflows into a retrieval-based paradigm, suggesting that dense visual features combined with expert-anchored text generation offer a viable path for scaling microscopy datasets beyond the limits of manual annotation.