🤖 machine learning

Self-Soupervision: Cooking Model Soups without Labels

This paper introduces "Self-Soupervision," a novel framework that extends model soups to self-supervised learning by allowing the combination of diverse SSL algorithms and hyperparameters to create more robust and accurate models without relying on labeled data.

Original authors: Anthony Fuller, James R. Green, Evan Shelhamer

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anthony Fuller, James R. Green, Evan Shelhamer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a master chef who has already learned to cook a basic, delicious soup (this is the "stock" or pre-trained model). Usually, if you want to make this soup even better, you might send the chef to a few different cooking schools to learn specific new techniques, then mix all those versions of the soup together into one final, super-delicious bowl. This is what the paper calls a "Model Soup."

However, in the past, these cooking schools required a strict menu of labeled ingredients (supervised learning). You had to tell the chef exactly what every vegetable was.

The Big Idea: Cooking Without a Recipe Book
This paper introduces a new method called "Self-Soupervision." The authors realized you don't need a labeled menu to make a great soup. Instead, you can let the chef taste the raw, unlabeled ingredients and figure out the flavors on their own (Self-Supervised Learning).

Here is how they "cook" these new soups, broken down into simple steps:

1. The Ingredients (The "Inter-Training")

Instead of just tweaking the recipe slightly, the authors send their "stock" model to four different, very different cooking schools.

School A teaches the model to reconstruct missing parts of a picture (like a puzzle).
School B teaches it to tell two different pictures apart.
School C and School D use other unique, unlabeled techniques.

Because these schools teach different skills without needing a teacher to say "yes, that's a cat," the resulting "ingredients" (the models) are very diverse. They have learned different things about the world just by looking at raw data.

2. The Mixing (The "Soup")

Usually, mixing different models is risky; it's like mixing a chocolate cake batter with a savory stew—it might taste terrible. But the authors found that when you mix these specific "Self-Soup" ingredients, they actually blend perfectly.

The Result: The final mixed soup is more robust and accurate than any single ingredient model on its own. It's like having a soup that tastes great whether it's raining, sunny, or if you accidentally dropped a little dirt in it (handling "corrupted" or messy data).

3. The Secret Sauce: "Self-Seasoning"

Normally, to pick the perfect mix of ingredients, you need a taste-tester with a list of correct answers (labels) to tell you which mix is best.

The Innovation: The authors invented "Self-Seasoning." This is a way to find the perfect mix without a taste-tester. The computer looks at the soup and asks, "Does this mix make sense?" It adjusts the recipe until the soup is as consistent and logical as possible, all without needing any labeled data.

Why This Matters (The "Taste Test")

The paper tested this recipe in a few specific ways:

Handling Messy Data: When they tested the soup on "corrupted" images (like photos with static noise or weird filters), the Self-Soup was significantly better than standard soups. It gained about 3.5% to 7% more accuracy on these messy tests.
Learning from New Environments: They showed that if you let the chef taste the type of food they will eventually serve (even without labels), the final soup is much better at handling that specific environment.
No Extra Labels Needed: They proved you can get results just as good as methods that do use labeled data, but without the expensive cost of labeling thousands of images.

The Bottom Line

Think of this paper as a new cookbook for AI chefs. It says: "You don't need a strict recipe book (labels) to make a world-class soup. If you let your chef explore different unlabeled ingredients and mix them together using our new 'Self-Seasoning' technique, you get a more robust, accurate, and versatile model."

The paper claims this works specifically for visual tasks (like recognizing images) and makes models better at handling real-world messiness, but it does not claim to solve medical diagnoses or other specific real-world applications outside of these image benchmarks.

Technical Summary: Self-Soupervision: Cooking Model Soups without Labels

1. Problem Statement

Model soups are a technique for improving prediction accuracy by averaging the parameters of multiple models (ingredients) initialized from a single pre-trained model (stock). While effective, existing soup methods rely entirely on supervised learning. They require labeled data to generate diverse ingredients, typically by varying optimization hyperparameters or fine-tuning on different labeled datasets. This dependency limits the "menu" of possible soups, preventing the utilization of the vast amounts of unlabeled data available in the real world. Furthermore, existing methods struggle to leverage distribution shifts or unlabeled target data to improve robustness without explicit labels.

The paper asks: Can we generalize model soups to Self-Supervised Learning (SSL)? Specifically, can we create diverse "ingredients" by training on unlabeled data using different SSL algorithms and losses, and then successfully mix them into a single robust model?

2. Methodology: Self-Soupervision

The authors introduce Self-Soupervision, a framework that generates model soup ingredients through independent self-supervised training runs before final supervised fine-tuning and mixing.

Core Framework

The process is defined formally as:
$\theta = \frac{1}{N \cdot M} \sum_{j=1}^{N} \sum_{i=1}^{M} \text{Train}(\underbrace{\text{Train}(\theta_{pt}, X_i)}_{\text{Self-Supervised Inter-training}}, X_j, Y_j)$
Where:

$\theta_{pt}$ is the pre-trained stock.
$M$ is the number of inter-training runs on unlabeled data $X_i$ using different SSL algorithms or hyperparameters.
$N$ is the number of fine-tuning runs on labeled data $X_j, Y_j$ for the target task.
The final model $\theta$ is the average of all resulting ingredients.

Key Components

Diverse SSL Ingredients: Instead of varying only supervised hyperparameters, the method varies the self-supervised training runs. Ingredients are created by inter-training on unlabeled data using different SSL families:
- Reconstruction: MAE (Masked Autoencoders).
- Instance-Contrastive: MoCoV3.
- Dimension-Contrastive: MMCR and LeJEPA.
- Hyperparameter Variation: Different learning rates and masking ratios within the same algorithm.
Shift-Aware Inter-training: To improve robustness, ingredients can be inter-trained on corrupted or shifted unlabeled data (e.g., ImageNet-C or LAION-C) before being fine-tuned back on the clean, labeled training distribution. This allows the model to "taste" the shift without requiring labels for that shift.
Self-Seasoning: Traditional soup mixing often uses "seasoning" (selecting the best mixture weights) based on a few-shot labeled set. The authors propose Self-Seasoning, an unsupervised variant. It optimizes mixture coefficients $\alpha$ by minimizing the entropy of nearest-neighbor predictions on unlabeled data, allowing for fully unsupervised soup creation for transfer tasks.

3. Key Contributions

Generalization to SSL: The paper demonstrates that model soups can be constructed from ingredients trained with different self-supervised losses and algorithms, not just supervised fine-tuning variations.
Linear Mode Connectivity (LMC) Discovery: The authors show for the first time that LMC—the condition where the linear interpolation of model weights maintains high accuracy—holds between ingredients trained with fundamentally different SSL algorithms (e.g., mixing MAE and MoCoV3 ingredients).
Robustness via Unlabeled Shifts: The method enables the creation of "shift-aware" ingredients by inter-training on corrupted unlabeled data, significantly boosting robustness on corrupted test sets without requiring labels for the corruption.
Fully Unsupervised Transfer: Through Self-Seasoning, the paper shows that soups can be mixed and optimized for new tasks using only unlabeled data, removing the need for task-specific labels during the mixing phase.

4. Experimental Results

The authors evaluated Self-Soupervision across ImageNet and the VTAB (21 diverse visual tasks) collections.

Robustness on Corrupted Data

ImageNet-C & LAION-C: Self-Soups that inter-train on corrupted unlabeled data (ImageNet-C/LAION-C) and then fine-tune on clean data achieved significant gains over standard supervised soups.
- +3.5% improvement on ImageNet-C.
- +7.0% improvement on LAION-C.
Comparison: These Self-Soups outperformed ensembles of individual ingredients and standard "Continued SSL + Supervised Soup" baselines (which inter-train on clean data only).

Transfer Learning (VTAB)

Clean Data: Self-Soups achieved modest but consistent gains over supervised-only soups and "Continued SSL" baselines across 21 VTAB tasks.
Corrupted Data: Self-Soups showed particular strength against blur, weather, and digital corruptions (+2% gain), though they performed slightly worse on noise compared to supervised-only soups.
Ratatouille Comparison: Self-Soups matched the performance of "Model Ratatouille" (which requires auxiliary labeled datasets for inter-training) while requiring no extra labeled datasets, proving the efficacy of using unlabeled data for ingredient diversity.

Self-Seasoning Performance

In few-shot scenarios (<500 training samples), Self-Seasoning (unsupervised mixture optimization) was competitive with "Labeled Seasoning" (supervised optimization) and outperformed standard fine-tuning on natural tasks. This demonstrates that high-quality soups can be created and tuned without task-specific labels.

5. Significance and Claims

The paper claims that Self-Soupervision "vastly expands the menu of possible soups" by harnessing unlabeled data and diverse SSL losses. Its primary significance lies in:

Democratizing Robustness: It allows practitioners to improve model robustness against distribution shifts using only unlabeled shifted data, bypassing the need for expensive labeled shift data.
Algorithmic Diversity: It proves that mixing models trained with different SSL objectives (e.g., reconstruction vs. contrastive) is not only possible but productive, challenging the assumption that ingredients must be homogeneous in their training objective.
Cost-Efficiency: The method improves accuracy and robustness without increasing inference costs (the final soup is a single model) and reduces the dependency on labeled data for the "cooking" process.

The authors conclude that while gains on clean data are modest, the ability to leverage unlabeled data for robustness and the ability to mix fundamentally different SSL algorithms represent a significant step forward in model merging and self-supervised learning.