Hierarchy-Guided Multimodal Representation Learning for Taxonomic Inference

This paper introduces CLiBD-HiR and CLiBD-HiR-Fuse, two hierarchy-aware multimodal learning frameworks that leverage the taxonomic structure of biological classification to significantly improve biodiversity identification accuracy and robustness against noisy or missing data compared to existing flat-label approaches.

The Gist

Imagine you are a detective trying to identify a mysterious animal found in the wild. You have two clues: a blurry photograph and a torn-up piece of its DNA. In the real world, nature is messy. Photos are often dark, blurry, or blocked by leaves. DNA samples are often incomplete or contain "typos" from the sequencing machine.

The goal of this paper is to build a super-smart AI detective that can identify animals (from broad categories like "Mammal" down to specific species like "Red Fox") even when these clues are imperfect.

Here is the breakdown of their solution, using some everyday analogies:

1. The Problem: The "Flat" vs. The "Family Tree"

Previous AI models treated animal names like a giant, flat list of random words. If the AI confused a "Red Fox" with a "Gray Wolf," it was just as wrong as confusing a "Fox" with a "Squid." It didn't understand that Foxes and Wolves are cousins (both are Canines), while Squids are totally different.

Because they didn't understand the Family Tree (the hierarchy of Order → Family → Genus → Species), when the clues were noisy or blurry, the AI would get completely lost. It might guess a completely unrelated animal instead of just guessing the wrong type of fox.

2. The Solution: Two New Tricks

The authors built on an existing AI called CLIBD (which already knows how to match photos, DNA, and text) and added two major upgrades:

Trick #1: The "Nested Doll" Rule (Hierarchical Information Regularization)

• The Analogy: Imagine Russian nesting dolls. The smallest doll is the specific species (e.g., Red Fox). Inside it is a slightly bigger doll (Genus: Foxes). Inside that is a Family doll (Canines), and so on.
• How it works: The new AI, called CLIBD-HiR, is forced to learn that the "Fox" doll must always fit inside the "Canine" doll.
• The Benefit: If the photo is so blurry that the AI can't tell if it's a Red Fox or a Gray Fox, it doesn't panic and guess "Squid." Because of the "Nested Doll" rule, it knows it's definitely a Fox. Even if it gets the specific species wrong, it stays correct at the broader levels (Genus, Family). This makes the AI much more robust against bad data.

Trick #2: The "Smart Translator" (Adaptive Fusion)

• The Analogy: Imagine you are trying to identify a suspect. Sometimes you only have a sketch (Image). Sometimes you only have a fingerprint (DNA). Sometimes you have both, but the sketch is smudged and the fingerprint is partial.
• How it works: The second version, CLIBD-HiR-Fuse, adds a "Smart Translator" module. Instead of just blindly mixing the photo and DNA together (like averaging two numbers), this module acts like a wise judge.
  • If the DNA is full of errors, the judge says, "Ignore the DNA, trust the photo more."
  • If the photo is too dark, it says, "Rely on the DNA."
  • If both are good, it combines them perfectly.
• The Benefit: This allows the system to work even if one of the clues is missing or broken, which happens constantly in real-world biodiversity research.

3. The Results: Why It Matters

The researchers tested this on a massive dataset of over 900,000 insect samples.

• The Score: Their new method improved accuracy by over 14% compared to previous state-of-the-art models.
• The Real-World Win: The biggest improvements happened when the data was "dirty" (blurry photos or corrupted DNA). In these messy scenarios, their AI was significantly better at saying, "I'm not 100% sure of the exact species, but I know it's definitely this type of beetle," rather than making a wild, incorrect guess.

Summary

Think of this paper as teaching an AI to think like a biologist rather than a robot.

1. It learns that biology is a hierarchy (like a family tree), so it doesn't get confused when details are fuzzy.
2. It learns to adapt to missing or broken clues, knowing when to trust the photo and when to trust the DNA.

This makes the AI a much more reliable tool for conservationists and scientists who need to identify species in the wild, where perfect data is a luxury they rarely get.

Technical Summary

1. Problem Statement

The paper addresses the challenge of robust taxonomic inference (predicting order, family, genus, or species) from large-scale, imperfect biodiversity data. Real-world inputs often suffer from:

• Modality Degradation: DNA barcodes may have partial reads, ambiguous bases, or sequencing artifacts; field images often suffer from blur, occlusion, and lighting variations.
• Missing Modalities: In practice, datasets often contain only images, only DNA, or both, with varying quality.
• Flat Label Limitations: Existing multimodal foundation models (e.g., CLIBD) treat taxonomy as a flat label space. They fail to encode the inherent biological hierarchy (Order $\to$ Family $\to$ Genus $\to$ Species). Consequently, noise can cause embeddings to drift arbitrarily, leading to catastrophic errors where a species prediction is wrong, but the higher-level family prediction is also incorrect.

2. Methodology

The authors propose CLiBD-HiR, a taxonomy-aware multimodal framework built upon the CLIBD architecture. The framework consists of two end-to-end variants and introduces a novel regularization technique.

A. Core Architecture

The model uses three encoders to map inputs into a shared embedding space:

• Image Encoder ($f_V$): Based on BioCLIP or OpenCLIP (ViT-L/14).
• DNA Encoder ($f_D$): Based on DNABERT2.
• Text Encoder ($f_T$): A frozen BioCLIP text encoder (to leverage strong biological priors).

B. Key Innovation: Hierarchical Information Regularization (HiR)

To enforce a hierarchy-consistent geometry, the authors introduce HiR, an image-only loss function inspired by HiConE.

• Mechanism: It treats samples sharing coarser labels (e.g., same Family) as positives at that level, and samples sharing finer labels (e.g., same Species) as positives at finer levels.
• Rectified Loss: Crucially, it enforces a constraint where finer-level losses cannot be optimized until coarser-level structures are established.
  • Mathematically, for level $\ell > 1$, the pair loss $\tilde{\ell}^{(\ell)}$ is clamped to be at least the maximum loss observed at the previous coarser level ($m^{(\ell-1)}$).
  • $\tilde{\ell}^{(\ell)}(i, j) = \max(\ell^{(\ell)}(i, j), m^{(\ell-1)})$.
• Effect: This prevents the model from overfitting to noisy fine-grained labels (species) if the coarse structure (genus/family) is not yet stable. It acts as a "noise stabilizer," ensuring that even if a species prediction fails due to noise, the embedding remains anchored to the correct higher-level taxonomic cluster.

C. Two Variants

1. CLiBD-HiR (Algo 1): Focuses on learning a structured, noise-robust embedding space using cross-modal contrastive losses (Image-Text, DNA-Text, Image-DNA) plus the HiR loss. Inference is performed via nearest-neighbor search in the embedding space.
2. CLiBD-HiR-Fuse (Algo 2): Adds a lightweight Gated Fusion predictor (a 2-layer MLP) trained jointly with the encoders.
   • It adaptively combines image and DNA embeddings ($z_i = [v_i; d_i]$) using a learned gate.
   • It supports Image-only, DNA-only, and Joint (Image+DNA) inference, making it resilient to missing or corrupted modalities.

3. Key Contributions

1. Hierarchical Information Regularization (HiR): A novel objective that explicitly shapes embedding geometry to respect biological hierarchy, significantly improving robustness against noisy and incomplete inputs.
2. Two End-to-End Variants:
   • CLiBD-HiR: Optimized for hierarchical prediction without explicit fusion.
   • CLiBD-HiR-Fuse: Introduces an adaptive fusion mechanism that outperforms naive averaging, particularly when modality quality varies.
3. Robustness to Real-World Noise: The framework is evaluated under realistic degradation scenarios (e.g., simulated sequencing errors, image blur) where existing baselines fail.

4. Experimental Results

The models were evaluated on the BIOSCAN-1M insect dataset (approx. 900k training samples, 224k test samples) across four taxonomic levels (Order, Family, Genus, Species).

• Performance Gains:
  • Compared to the strong baseline (CLIBD), CLiBD-HiR improved Global Top-1 accuracy by >14% under noisy conditions.
  • DNA Robustness: Under noisy DNA conditions, CLiBD-HiR improved Global Top-1 from 52.4% (CLIBD) to 66.0%.
  • Image Robustness: Under noisy image conditions, Global Top-1 improved from 40.0% to 46.6%.
• Fusion Benefits:
  • The CLiBD-HiR-Fuse variant (Algo 2) significantly outperformed naive embedding averaging when both modalities were noisy.
  • In the "Noisy I+D" scenario, the learned fusion achieved 88.0% Global Top-1 accuracy, compared to 85.5% for simple averaging.
  • The largest gains were observed at the Species level under joint noise (57.4% vs 54.6%).
• Hierarchy Preservation: The results confirmed that HiR ensures predictions remain correct at coarser levels (Genus/Family) even when species-level predictions are incorrect due to noise.

5. Significance

This work demonstrates that explicitly encoding biological hierarchy is critical for building practical biodiversity foundation models.

• Scientific Impact: It bridges the gap between curated research datasets and messy, real-world operational data (e.g., environmental monitoring, conservation).
• Methodological Impact: It moves beyond standard contrastive learning by introducing a hierarchy-aware regularization that prevents semantic drift in noisy environments.
• Practical Utility: The adaptive fusion capability allows the model to function effectively in diverse field scenarios where data availability is inconsistent, making it a robust tool for automated biodiversity identification.