TaxaAdapter: Vision Taxonomy Models are Key to Fine-grained Image Generation over the Tree of Life

1. Problem Statement

Generating high-fidelity images of specific biological species is a critical challenge in biodiversity science and computer vision. While state-of-the-art text-to-image (T2I) diffusion models (e.g., Stable Diffusion, FLUX) excel at generating realistic images of broad categories (e.g., "a bird"), they fail at fine-grained species synthesis.

• Subtle Visual Traits: Species identity is often defined by minute morphological differences (e.g., beak curvature, plumage patterns) that general models cannot distinguish.
• Long-Tailed Distribution: Biodiversity datasets are extremely long-tailed; many of the ~10 million species on Earth have very few or no training images.
• Limitations of Existing Methods:
  • Standard T2I Models: Rely on coarse semantic categories and lack the hierarchical taxonomic knowledge to distinguish closely related species.
  • Taxonomy-Aware Models (e.g., TaxaDiffusion): Often require training taxonomic embeddings from scratch, which is computationally expensive and limits generalization to unseen species (Out-of-Distribution or OOD).
  • Text Encoders: Standard encoders (like CLIP) are trained on general web data and lack the specific <species name, image> alignment needed for biological precision.

2. Methodology: TaxaAdapter

The authors propose TaxaAdapter, a lightweight, plug-in framework that injects Vision Taxonomy Model (VTM) embeddings into a frozen text-to-image diffusion model.

Core Architecture

• Dual Conditioning Mechanism: The model utilizes two parallel conditioning streams:
  1. Text Stream: Uses a frozen CLIP text encoder to process natural language prompts (e.g., "a photo of a bird on a tree"), controlling context, style, pose, and background.
  2. Taxonomy Stream: Uses a pre-trained VTM (e.g., BioCLIP, TaxaBind, or BioTrove-CLIP) to encode the full taxonomic hierarchy (Kingdom $\to$ Species) into an embedding. This provides species-specific morphological knowledge.
• Decoupled Cross-Attention: Instead of concatenating or averaging embeddings, TaxaAdapter employs a decoupled cross-attention mechanism.
  • The U-Net features (Query) attend separately to the Text Key/Value pairs and the Taxonomy Key/Value pairs.
  • This allows the model to learn distinct roles: the taxonomy branch focuses on species-level morphology, while the text branch manages contextual attributes.
• Parameter-Efficient Training (PEFT):
  • The backbone diffusion model (e.g., Stable Diffusion v1.5) and the VTM encoders are frozen.
  • Only a lightweight projection network (for the taxonomy embedding) and the new cross-attention layers are trained.
  • This adds only ~22M trainable parameters (vs. ~860M in the full model), making training efficient and preventing catastrophic forgetting of the base model's text-to-image capabilities.

Inference Control

• A weighting factor ($\lambda$) modulates the strength of the taxonomy conditioning during inference.
  • $\lambda = 0$: Pure text-to-image generation (standard behavior).
  • $\lambda = 1$: Full taxonomy guidance (maximizing species fidelity).
  • Intermediate values allow balancing between biological accuracy and creative text control.

3. Key Contributions

1. TaxaAdapter Framework: The first method to successfully bridge pre-trained Vision Taxonomy Models with generative diffusion models without retraining the VTM or the diffusion backbone from scratch.
2. Dual Conditioning Strategy: A novel architecture that separates species identity (via VTM) from contextual control (via text), enabling both high-fidelity species synthesis and flexible free-form prompting.
3. New Evaluation Metric (MLLM-based Trait Fidelity):
   • Recognizing that standard metrics (FID, LPIPS) fail to capture morphological correctness, the authors introduce a Multimodal Large Language Model (MLLM) pipeline.
   • Process: (1) Generate trait captions for real and generated images using an MLLM (InternVL3); (2) Summarize these into a single species-level trait description using an LLM; (3) Compute text similarity (BERTScore, CLIPScore, etc.) between the real and generated summaries.
   • This provides an interpretable, trait-level measure of morphological consistency.
4. Strong Generalization: Demonstrated ability to generate accurate images for few-shot species (1-5 training images) and unseen species (OOD) by leveraging the pre-trained VTM's hierarchical knowledge.

4. Experimental Results

The authors evaluated TaxaAdapter on three biodiversity datasets: iNat-mini (500k images, 10k species), TreeOfLife-1M (1M images, 200k+ species), and FishNet (95k fish species).

• Quantitative Performance:
  • Species Identity: TaxaAdapter achieved a Classification Accuracy Score (CAS@1) of ~60-83% on iNat-mini and TreeOfLife-1M, significantly outperforming baselines (SD 1.5, SD 3.5, FLUX.1-dev) which scored <13% even with LoRA fine-tuning.
  • Taxonomy Alignment: It achieved the highest BioCLIP scores (e.g., 30.44 vs. 15.52 for SD 1.5), indicating superior alignment between the generated image and the taxonomic name.
  • Trait Fidelity: In the MLLM-based evaluation, TaxaAdapter consistently scored highest on BERTScore, CLIPScore, and ROUGE-L, proving it captures specific morphological traits better than baselines.
• Generalization Capabilities:
  • Few-Shot: Outperformed baselines on species with only 1 or <5 training images.
  • Out-of-Distribution (OOD): Successfully generated accurate images for 51 bird species from the CUB-200 dataset that were never seen during training, demonstrating the power of transferring VTM knowledge.
• Ablation Studies:
  • Removing the text branch or using a single attention stream significantly degraded performance, confirming the necessity of the dual-conditioning and decoupled attention design.
  • Using generic CLIP encoders instead of VTMs resulted in poor performance, highlighting the need for domain-specific taxonomy-image alignment.

5. Significance and Impact

• Scalability: TaxaAdapter offers a scalable solution for generating images across the entire Tree of Life, addressing the long-tail problem of biodiversity data where training data is scarce.
• Scientific Utility: The ability to generate morphologically accurate images of rare or unseen species aids biologists in visualizing diagnostic traits, creating educational materials, and augmenting datasets for downstream tasks.
• Paradigm Shift: The paper argues that for fine-grained generation in specialized domains, incorporating pre-trained domain-specific embeddings (VTMs) is more effective and efficient than training new embeddings from scratch or relying solely on general-purpose text encoders.
• Future Directions: The framework opens avenues for "trait discovery" (analyzing attention maps to understand taxonomic features) and potential applications in synthetic biology or gene-editing simulations (via hybrid species generation).

In conclusion, TaxaAdapter demonstrates that leveraging pre-trained Vision Taxonomy Models as a guiding signal is the key to unlocking scalable, fine-grained, and morphologically faithful image generation for the natural world.