Technical Summary: FINO – Adapting Vision Foundation Models with Metadata

Problem Statement

Visual foundation models, while powerful on natural images, often degrade when applied to specialized scientific domains such as biomedical microscopy, Earth observation, wildlife monitoring, and medical imaging. These domains exhibit substantial distribution shifts driven by both meaningful variations (e.g., geography, staining protocols) and spurious factors (e.g., sensor characteristics, batch effects).

Standard adaptation strategies face significant limitations in these settings:

• Supervised Fine-Tuning: Requires scarce or expensive task-specific labels. It often overfits to task signals, eroding the model's generalization capabilities and leading to poor out-of-distribution (OOD) performance.
• Unsupervised Domain Adaptation (UDA): Leverages unlabelled target data but remains task-driven, risking task-specific collapse and failing to generalize across diverse scientific shifts.
• Pure Self-Supervised Learning (SSL): While label-free, standard SSL adaptation can remain sensitive to low-level cues or context-specific artifacts, failing to distinguish between informative semantic factors and spurious acquisition noise.

Scientific datasets, however, are typically accompanied by rich metadata (e.g., antibody types, country, timestamp, viewing angle) that correlates with these variations. The challenge is to leverage this metadata to adapt representations without task labels, preserving informative structure while suppressing spurious factors.

Methodology: FINO (FIne tuning with NO labels)

The authors propose FINO, a unified framework for metadata-guided representation adaptation. The method adapts a pretrained foundation model to a new domain using only source data and its associated metadata, without requiring task labels, target distribution data, or metadata at inference time.

Core Architecture

FINO builds upon the standard DINO self-supervised setup (student-teacher architecture with exponential moving average updates) and extends it with metadata guidance.

1. Metadata Partitioning: Metadata types ($t$) are categorized into:
   • Informative Factors ($M^+$): Variables the model should preserve (e.g., antibody identity in microscopy, geography in Earth observation).
   • Spurious Factors ($M^-$): Variables the model should suppress (e.g., plate IDs, sensor resolution, batch effects).
2. Metadata Guidance Losses:
   • Discrete Metadata: For categorical metadata (e.g., country, antibody), FINO employs a prototype-based contrastive objective. It maintains a bank of prototypes $\{p_m\}$ for each metadata value. The student embedding is contrasted against the prototype corresponding to its metadata value. Prototypes are updated via an EMA of teacher embeddings. This approach scales to high-cardinality metadata without requiring positive/negative pairs within a batch.
   • Continuous Metadata: For continuous values (e.g., timestamp, viewing angle), a learnable predictor (MLP) is trained to regress the metadata value from the embedding, minimizing a regression loss.
3. Gradient Reversal for Spurious Factors: To suppress spurious factors, FINO applies gradient reversal to the metadata loss before it reaches the encoder. This encourages the encoder to ignore these factors while the predictor head still learns to predict them (adversarial formulation).
4. Overall Objective: The total loss combines standard self-supervised losses (DINO on CLS tokens, iBOT on patch tokens), a regularization term (SIGReg), and the weighted sum of metadata guidance losses:
   $$L(x) = L_{DINO}(x) + L_{iBOT}(x) + \lambda_{SIGReg}L_{SIGReg}(x) + \sum_{t \in M^+ \cup M^-} \lambda^{(t)}_{meta} L^{(t)}_{meta}(x)$$
   Notably, the method uses a single, fixed hyperparameter ($\lambda_{meta} = 0.03$) across all domains, requiring no extensive tuning.

Training Protocol

• Phase 1: The pretrained backbone is frozen. Metadata heads and loss heads are initialized randomly and trained to align with the pretrained representation space. The patch embedding layer is also updated to adapt to target domain pixel statistics.
• Phase 2: The backbone is progressively unfrozen and trained end-to-end with a linear learning rate warmup to prevent destabilization.

Key Contributions

1. Unified Framework: FINO integrates heterogeneous metadata (discrete and continuous) into self-supervised learning without modifying the backbone architecture or requiring metadata at inference.
2. Label-Free Adaptation: It demonstrates that a generic vision foundation model can be specialized to scientific domains using only metadata, outperforming fully supervised fine-tuning which relies on task labels.
3. Robustness and Transfer: The method produces representations that are robust to distribution shifts and task-agnostic, enabling effective transfer across datasets within the same domain (e.g., from HPA to OpenCell) even when class definitions differ.
4. State-of-the-Art Performance: FINO matches or exceeds highly specialized, domain-specific state-of-the-art methods (including top Kaggle solutions) across four distinct scientific domains using a single shared recipe.

Experimental Results

The authors evaluated FINO on four datasets: Human Protein Atlas (HPA), iWildCam, Functional Map of the World (FMoW), and MIMIC-CXR.

• Performance vs. Supervised Fine-Tuning: FINO consistently outperformed fully supervised fine-tuning (which trains the backbone and a probe) on OOD metrics. For instance, on HPA, FINO achieved an F1 of 61.2, surpassing supervised fine-tuning (52.6) and specialized SOTA (59.4), despite using no task labels for backbone adaptation.
• Performance vs. SSL and UDA: FINO surpassed standard self-supervised adaptation (DINO) and unsupervised domain adaptation methods (CORAL, DANN), which often failed to generalize or collapsed on specific datasets like iWildCam.
• Low-Data Regime: In scenarios with only 1–10% of labels available for the downstream probe, FINO maintained high performance (50.9–57.0%), whereas supervised fine-tuning degraded sharply.
• Representation Analysis: UMAP visualizations on HPA showed that FINO organized features according to the target variable (protein localization) rather than spurious acquisition factors (cell line), achieving a k-NN accuracy of 83% on localization versus 75% on cell line (compared to 87% on cell line for standard SSL).
• Transfer Learning: FINO adapted models successfully transferred to unseen datasets within the same domain (e.g., HPA $\to$ OpenCell), outperforming both frozen baselines and supervised adaptation.

Significance and Claims

The paper posits that scientific application domains are a critical and fruitful setting for evaluating computer vision foundation models. The authors claim that:

• Metadata is a powerful, underutilized resource: It serves as effective weak supervision to steer self-supervised learning toward robust, domain-adapted representations.
• Task-agnostic adaptation is superior: By decoupling representation learning from specific task labels, FINO avoids the "collapse" of generality often seen in supervised fine-tuning, resulting in features that are both more robust to distribution shifts and more aligned with downstream tasks.
• Scalability: The method handles highly granular metadata (thousands of discrete categories or continuous values) without the need for complex domain-specific engineering or extensive hyperparameter tuning.

The work concludes that FINO establishes a new paradigm where foundation models can be specialized for science using the "metadata you already have," achieving performance that rivals or exceeds specialized solutions without the burden of task-specific labeling.