💻 computer science

SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Models

The paper introduces SigLino, an efficient family of agglomerative vision foundation models that leverages asymmetric distillation, token-balanced batching, and hierarchical data sampling to effectively distill knowledge from SigLIP2 and DINOv3 into dense and Mixture-of-Experts students, resulting in superior performance for early-fusion Grounding-VLMs compared to models trained from scratch.

Original authors: Sofian Chaybouti, Sanath Narayan, Yasser Dahou, Phúc H. Lê Khac, Ankit Singh, Ngoc Dung Huynh, Wamiq Reyaz Para, Hilde Kuehne, Hakim Hacid

Published 2026-04-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sofian Chaybouti, Sanath Narayan, Yasser Dahou, Phúc H. Lê Khac, Ankit Singh, Ngoc Dung Huynh, Wamiq Reyaz Para, Hilde Kuehne, Hakim Hacid

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a young, talented artist (the Student) how to see the world perfectly. Usually, you'd hire one master painter to teach them. But what if you could hire two different masters, each with a unique superpower, and have them teach the student simultaneously?

That is exactly what the SigLino paper does. It introduces a new way to train AI vision models by combining the strengths of two different "Master Teachers" into one super-efficient student.

Here is the breakdown of how they did it, using simple analogies:

1. The Two Masters (The Teachers)

The researchers didn't just pick any teachers; they picked two specific "Masters" who are experts at different things:

Master SigLIP2: This teacher is great at language. It knows exactly what an image is called and can describe it in words. It's like a poet who can look at a picture and write a perfect caption.
Master DINOv3: This teacher is great at geometry and details. It understands the shape, texture, and boundaries of objects perfectly. It's like a sculptor who knows exactly where every edge of a statue is.

The Problem: Usually, if you try to learn from both, the student gets confused. The poet wants to talk about words, and the sculptor wants to talk about shapes. They speak different languages, and the student struggles to learn from both at once without getting overwhelmed.

2. The Solution: SigLino (The Student)

SigLino is a new "Student" model designed to listen to both masters at the same time without getting a headache. It uses a special architecture called a Mixture-of-Experts (MoE).

The Analogy: Imagine a classroom with one teacher (SigLIP2) and another (DINOv3). Instead of having one student try to take notes on everything, the student has a team of specialized assistants (the "Experts").

When the class discusses poetry, the "Poetry Assistant" takes notes.
When the class discusses shapes, the "Sculpting Assistant" takes notes.
They all work together to form one complete understanding.

This allows the student to learn from both masters efficiently, using fewer computer resources than previous methods.

3. The Secret Sauce: Three Magic Tricks

The paper reveals three specific "tricks" that made this learning process so much faster and better:

A. The "Balanced Lunch" (Token-Balanced Batching)

The Problem: Imagine a classroom where some students bring a tiny sandwich (small, low-resolution images) and others bring a massive feast (huge, high-resolution images). If you try to feed everyone at once, the table gets messy, and the students with tiny sandwiches get ignored because the big feast takes up all the space.
The Fix: The researchers invented a way to cut the big feasts into smaller, bite-sized pieces and pack them together with the tiny sandwiches so that every student gets the exact same amount of food (tokens). This keeps the learning stable and ensures the AI doesn't forget how to see small details just because it's looking at big pictures.

B. The "Group Hug" (Asymmetric Relational Knowledge Distillation)

The Problem: Usually, AI learns by comparing one image to its teacher. But sometimes, the teacher makes a mistake, or the student gets confused about how two different images relate to each other.
The Fix: The researchers taught the student to look at the relationships between images.

Normal Learning: "This image looks like a cat."
SigLino Learning: "This image is a cat, and that image is a dog. They are different. But these two images of cats are very similar."
The "Asymmetric" Twist: They added a rule: "Only push images together if they are already close, and only pull them apart if they are already far." This prevents the AI from getting confused and forcing unrelated things to look alike. It's like a dance instructor who only tells you to hold hands with people you are already standing next to, rather than forcing you to grab strangers across the room.

C. The "Curated Library" (OpenLVD200M)

The Problem: The internet is full of garbage data. If you teach a student by letting them read every book in a messy library, they will learn a lot of nonsense.
The Fix: The researchers built a special library called OpenLVD200M. Instead of grabbing 200 million random images, they used a smart sorting system (hierarchical clustering) to pick images that cover every topic evenly.

Analogy: Instead of grabbing 200 million random pages from a library, they carefully selected one page from every single book to ensure the student learns about "apples," "cars," "clouds," and "microscopes" equally, without getting bored by seeing 1,000 pictures of the same cat.

4. The Result: A Super-Student

The result is a model that is:

Faster: It learns with less data and less computing power.
Smarter: It understands both words and shapes better than models trained from scratch.
Versatile: It can be used to build "Grounding VLMs"—AI systems that can not only describe an image but also point to exactly where an object is (like circling a specific bird in a photo).

Summary

SigLino is like a super-efficient apprenticeship program. By carefully organizing the "classroom" (batching), teaching the student to understand relationships between objects (relational distillation), and curating the best possible textbooks (OpenLVD200M), they created an AI that learns from two different masters simultaneously, becoming a master of both language and vision in record time.

1. Problem Statement

Vision Foundation Models (VFMs) typically follow one of two paradigms:

Modular Vision-Language Models (VLMs): Pairing a text-aligned vision encoder with a Large Language Model (LLM). While effective for instruction following, they often struggle with dense prediction tasks (e.g., segmentation) and are not natively multi-modal.
Specialized Foundation Models: Trained on single supervision sources (e.g., purely visual or purely image-text). These excel at their specific objective but lack the generalization depth required for unified vision-language understanding.

Agglomerative VFMs attempt to unify these capabilities by distilling knowledge from multiple teacher models into a single student backbone. However, existing approaches (e.g., AM-RADIO, RADIOv2.5) face significant challenges:

Computational Cost: They require massive datasets and expensive training.
Training Instability: Handling varying image resolutions and multiple loss functions often leads to unstable optimization or catastrophic forgetting of specific teacher capabilities.
Data Efficiency: Random sampling of web-scraped data often fails to capture fine-grained concepts effectively.
Geometric Preservation: Standard distillation often fails to preserve the pairwise geometric relationships (structure) of teacher embeddings, which is crucial for tasks like image-text alignment and dense prediction.

2. Methodology: SigLino

The authors propose SigLino, an efficient family of agglomerative vision foundation models that distills knowledge simultaneously from SigLIP2 (strong image-text alignment) and DINOv3 (strong dense visual understanding and geometric coherence).

Key Technical Components:

A. Architecture: Mixture-of-Experts (MoE)

The student model utilizes an MoE architecture with modality-specific experts. This allows the model to naturally accommodate complementary signals from different teachers (e.g., text-aligned features vs. dense geometric features) without interference.
The student includes Register tokens (similar to DINOv3) and CLS tokens.
Teacher-Specific Heads: Learnable projection heads map student features into the embedding spaces of SigLIP2 and DINOv3. For SigLIP2, a frozen attention pooling layer is reused to compute global summaries, ensuring the student respects the teacher's representation logic.

B. Asymmetric Relational Knowledge Distillation (ARKD)

Standard Relational KD (RKD) aligns pairwise distances between samples but can degrade clustering quality (kNN performance) by aggressively pushing/pulling samples.
ARKD introduces an asymmetric constraint:
- It only penalizes the student if it expands the distance between two samples that are close in the teacher space, or shrinks the distance between samples that are far apart.
- This acts as a regularizer that preserves the teacher's geometric structure (crucial for DINOv3's dense features) while accelerating image-text alignment, without sacrificing clustering quality.

C. Token-Balanced Batching

Training on native-resolution images creates highly variable token counts per image (e.g., $256\times256$ vs. $768\times768$ ), leading to unbalanced GPU loads and unstable gradients.
Solution: The authors pack multiple images of varying resolutions into a single sequence up to a fixed token budget using FlexAttention.
Per-Image Loss Normalization: To prevent high-resolution images from dominating the gradient, losses are normalized by the number of tokens per image before global aggregation. This stabilizes training across resolutions and prevents "low-resolution forgetting."

D. Data Curation: OpenLVD200M

Instead of random sampling from web-scraped data (LAION/DFN), the authors use hierarchical clustering to create OpenLVD200M (200M images).
This method ensures uniform coverage of visual concepts, mitigating long-tail biases and significantly improving sample efficiency, particularly for fine-grained recognition.

E. Early-Fusion Grounding VLM Initialization

The distilled SigLino model is used to initialize the vision experts of an Early-Fusion Grounding VLM.
To prevent the degradation of dense feature maps during VLM fine-tuning (a common issue where global alignment harms spatial coherence), the authors employ Gram Anchoring. This loss function preserves the Gram matrix (spatial structure) of the frozen distilled teacher features during adaptation.

3. Key Contributions

OpenLVD200M Dataset: A curated 200M-image dataset constructed via hierarchical clustering, demonstrating superior efficiency for multi-teacher distillation compared to random sampling.
ARKD Loss: A novel asymmetric relational distillation loss that improves image-text alignment (especially for DINOv3) while preserving clustering properties.
Token-Balanced Batching: A training strategy enabling stable, efficient multi-resolution training without performance sacrifice.
MoE Agglomerative Model: A demonstration that MoE architectures naturally facilitate early-fusion grounding VLMs, achieving strong performance with limited annotations.
Open Source: Release of the OpenLVD200M dataset and five distilled checkpoints (Dense and MoE variants).

4. Results

The authors evaluated SigLino against state-of-the-art baselines (RADIOv2.5, TIPS) and teacher models (SigLIP2, DINOv3) across multiple benchmarks:

Classification & Retrieval: SigLino-MoE achieves State-of-the-Art (SOTA) performance on global representation benchmarks (ImageNet, CUB-200, etc.) and retrieval tasks (MSCOCO, Flickr30k). Notably, it outperforms RADIOv2.5-H on macro-averaged image-text classification (84.13 vs. 82.26) and kNN (88.06 vs. 85.12) while using 4.7x fewer training tokens (0.23T vs. 1.1T).
Dense Prediction: The models show competitive results on segmentation tasks (ADE20k, Cityscapes) via linear probing, outperforming RADIOv2.5 variants.
Grounding VLM: Initializing an early-fusion VLM with SigLino yields massive gains over training from scratch.
- Referring Expression Segmentation (RES): SigLino initialization achieves 54.72 on RefCOCO, significantly outperforming single-teacher initialization (SigLIP2: 40.69, DINOv3: 45.06).
- Gram Anchoring: Adding Gram anchoring during VLM fine-tuning further improves performance, preserving spatial coherence.
Efficiency: The MoE variant (0.15B active parameters) outperforms the dense RADIOv2.5-H (0.6B), demonstrating an excellent efficiency-performance trade-off.

5. Significance

Efficiency: SigLino proves that high-quality, unified vision representations can be learned with significantly less data and compute than previous agglomerative models, making advanced VFMs more accessible.
Unified Paradigm: It successfully bridges the gap between dense visual understanding (DINOv3) and language alignment (SigLIP2) in a single backbone, enabling robust early-fusion VLMs that do not rely on modular stacks.
Methodological Advances: The introduction of ARKD and token-balanced batching provides new best practices for multi-teacher distillation, addressing critical issues of geometric preservation and training stability at scale.
Practical Impact: The release of OpenLVD200M and the distilled models offers a strong foundation for future research in efficient, multi-modal, and grounded vision systems.