Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols

This paper introduces FEWTRANS, a comprehensive benchmark and the Hyperparameter Ensemble (HPE) evaluation protocol to rigorously assess few-shot transfer learning, revealing that pre-trained model selection and full-parameter fine-tuning often outperform sophisticated adaptation methods due to their ability to make distributed micro-adjustments without overfitting.

The Gist

Imagine you have a brilliant chef who has spent years cooking in a massive, world-famous kitchen (this is the Pre-trained Model). They know how to make thousands of dishes perfectly. Now, you want to hire this chef to cook a very specific, rare dish for a small dinner party, but you only have three ingredients to work with (this is Few-Shot Learning).

The big question in the tech world right now is: How do we best teach this chef to cook this new dish with so little information?

This paper, FEWTRANS, is like a new, super-strict food critic who says, "Stop guessing! We need a fair way to test these chefs." Here is the breakdown of their findings in simple terms:

1. The Problem: The "Lucky Draw" and The "Fake Test"

The authors found that previous ways of testing these AI chefs were flawed in two big ways:

• The "Lucky Draw" (Sampling Lottery): Imagine you ask the chef to cook a dish using three random ingredients. If you happen to pick three ingredients that happen to go well together, the chef looks like a genius. If you pick three weird ones, they look terrible. Previous studies often only tested the chef once or twice. If they got lucky, they got a high score. The authors say, "We need to test them 600 times with different random ingredients to see their true skill."
• The "Fake Test" (Validation Set Illusion): In the real world, you don't have extra ingredients to practice on before the party. But previous tests let the chef practice on a huge pile of extra ingredients to figure out the perfect cooking temperature. The authors say, "That's cheating! In the real world, you have to guess the temperature based on just the three ingredients you have."

2. The Solution: The "Swarm of Chefs" (Hyperparameter Ensemble)

To fix the "Fake Test" problem without giving the chef extra ingredients, the authors invented a new protocol called HPE (Hyperparameter Ensemble).

Instead of asking the chef to pick one perfect temperature and time, they say: "Let's have 9 different versions of the chef try the dish at 9 different temperatures and times all at once. Then, we take the average of their results."

• Why this is smart: If a method is "volatile" (it works great at one temperature but fails miserably at another), the average score will be low. If a method is "robust" (it works well across many temperatures), the average score will be high. This acts like a safety net, punishing unreliable methods and rewarding stable ones.

3. The Big Surprise: The "Simple Chef" Wins

The researchers tested many fancy, complex algorithms designed to be "efficient" (like only changing a few spices instead of the whole recipe). They expected these fancy methods to win.

They didn't.

The winner was the Simple Full Fine-Tuning method. This is like telling the chef: "Go ahead, change everything in your recipe if you need to, even if you only have three ingredients."

• The Result: The simple method actually performed better than the fancy, restricted methods.
• Why? The authors discovered that the simple method doesn't go crazy. Instead, it makes tiny, distributed "micro-adjustments" to the whole recipe. It's like gently nudging the entire kitchen to fit the new dish, rather than trying to force a specific part of the kitchen to do all the work. This keeps the chef from "overfitting" (getting too obsessed with those three specific ingredients and forgetting how to cook generally).

4. The "Language Barrier" Problem

The paper also looked at Multimodal Models (AI that understands both pictures and words, like CLIP).

They found that these models struggle when the names of the objects are rare or scientific.

• Example: If the dish is "Mushroom A," the AI knows what a mushroom is. But if the dish is "Agaricus cupreobrunneus" (a specific Latin name for a mushroom), the AI gets confused because it has never seen that word in its training.
• The Fix: The simple "change everything" method (Full Fine-Tuning) is the only thing that fixes this. It forces the AI to re-learn the connection between the picture and the weird word, acting as a translator that bridges the gap.

5. The Takeaway

The authors built a new "Ruler" called FEWTRANS to measure AI performance fairly. Their main message is:

1. Stop overcomplicating things. The most important factor for success is which model you start with (the chef's raw talent), not the fancy algorithm you use to adapt it.
2. Simple is often better. Just letting the model adjust everything slightly often works better than trying to be clever and only changing a few parts.
3. Be realistic. Don't test AI in a lab with unlimited data; test it in the messy, data-scarce reality where it will actually be used.

In short: Don't trust the hype of complex algorithms. Sometimes, the old-school method of just "learning from scratch with what you have" is still the most powerful tool in the box.

Technical Summary

1. Problem Statement

The paper addresses critical flaws in the current evaluation of few-shot transfer learning for pre-trained models (both vision-only and multimodal). Despite the proliferation of sophisticated transfer algorithms (e.g., LoRA, Adapters, Prompt Tuning), the field lacks a unified, rigorous evaluation protocol. The authors identify two primary deficiencies in existing literature:

• The "Sampling Lottery" Effect: Previous evaluations often rely on a small number of randomly sampled tasks (e.g., 3 tasks). In few-shot regimes, performance is highly volatile due to random sampling; a single sample change can drastically alter results, making comparisons based on few tasks unreliable.
• The "Validation Set Illusion": Standard hyperparameter tuning relies on large, held-out validation sets from the target domain. In real-world few-shot scenarios, such data is unavailable. Tuning on large validation sets or using empirical defaults leads to unrealistic performance estimates that do not reflect true deployment capabilities.

Additionally, the authors note that existing benchmarks often lack diversity, use class-balanced settings (unrealistic for real-world data), and include datasets with known label noise or saturation issues.

2. Methodology: The FEWTRANS Benchmark

To address these issues, the authors introduce FEWTRANS, a comprehensive benchmark and evaluation framework.

A. Dataset Construction

FEWTRANS comprises 10 diverse downstream datasets (e.g., ImageNet-Sketch, EuroSAT, Fungi, Plant Disease, Aircraft) selected from an initial pool of 40+ candidates.

• Selection Criteria: Datasets were filtered for multimodal compatibility (natural language class names), domain diversity, difficulty (excluding "easy" saturated tasks), and data integrity (removing datasets with high noise like Stanford Cars).
• Task Sampling: Unlike previous works, FEWTRANS samples 600 tasks per dataset. Tasks vary in the number of classes (2–15) and shots (1–100), including class-imbalanced scenarios to mimic real-world conditions.
• Base-Novel Split: For multimodal models, a 4:1 base-to-novel split is used to evaluate generalization to unseen classes.

B. The Hyperparameter Ensemble (HPE) Protocol

To overcome the "validation set illusion" without requiring extra labels, the authors propose the Hyperparameter Ensemble (HPE) protocol:

• Mechanism: Instead of searching for a single "best" hyperparameter configuration (which is impossible without a validation set), HPE aggregates predictions from a grid of $N$ hyperparameter configurations (e.g., varying learning rates and epochs).
• Fusion: The final prediction is the average of logits from all configurations in the grid.
• Benefits:
  • Robustness: It approximates the performance of an "Oracle" (best possible configuration) without explicit validation.
  • Sensitivity Penalty: It naturally penalizes volatile methods. If a method's performance fluctuates wildly with hyperparameter changes, the ensemble average drops, reflecting the risk of deployment in real-world settings where tuning is impossible.
  • Fairness: All methods are evaluated under the same grid constraints, ensuring a fair comparison of intrinsic robustness.

3. Key Contributions

1. FEWTRANS Benchmark: A rigorous, large-scale benchmark with 10 diverse datasets and 6,000 sampled tasks, eliminating the "sampling lottery" through statistical significance (95% confidence intervals).
2. HPE Protocol: A novel, label-free evaluation method that provides a realistic estimate of model performance and sensitivity to hyperparameters in data-scarce regimes.
3. Mechanistic Insights:
   • Full Fine-Tuning (Full-FT) Superiority: Demonstrating that simple full-parameter fine-tuning often outperforms or matches sophisticated Parameter-Efficient Fine-Tuning (PEFT) methods.
   • Micro-adjustment Theory: Explaining why Full-FT works without overfitting (via distributed micro-adjustments).
   • Linguistic Domain Shift: Quantifying the failure of multimodal models in specialized domains due to rare vocabulary.

4. Key Results and Findings

A. Evaluation of Transfer Algorithms

• Full-FT vs. PEFT: Contrary to the trend of developing complex PEFT methods (LoRA, Adapters, Prompt Tuning), Full-FT consistently achieves the best or comparable performance across vision and multimodal models.
• Negligible Gaps: Statistical analysis (paired t-tests on 6,000 tasks) reveals that the performance difference between Full-FT and sophisticated methods (like LoRA) is often negligible (Cohen's $d \approx -0.22$, a "small" effect size). On many datasets, there is no statistically significant difference.
• Multimodal Failure in Specialized Domains: Multimodal models (like CLIP) suffer significant performance collapse on datasets with rare class names (e.g., Fungi, Plant Disease). In these cases, Full-FT of the visual encoder alone often outperforms methods that also tune the text encoder or use prompts.

B. Mechanistic Analysis

• Parameter Update Scales: Full-FT succeeds via distributed micro-adjustments (L2 norm of weight changes $\Delta W$ is very small, 0.01–0.07). This keeps the model within the "flat minima" basin of the pre-trained weights, acting as implicit regularization and preventing overfitting. In contrast, PEFT methods often induce larger shifts that may lead to "sharp" minima.
• Feature Reshaping: Full-FT allows for more flexible reshaping of high-level semantic representations (lower CKA similarity in deep layers) compared to the constrained adaptation of PEFT methods.
• Text Domain Shift: The paper quantifies the "text semantic rarity" using Adjusted Zipf Frequency Scores. A strong negative correlation ($\rho = -0.881$) exists between text rarity and few-shot adaptation gain. When class names are rare (e.g., scientific Latin names), the text encoder fails to provide semantic anchors, necessitating Full-FT to realign visual features directly.

5. Significance and Impact

• Re-evaluating Progress: The paper suggests that the field may be stagnating in terms of algorithmic innovation for few-shot transfer. The "sophisticated" methods offer little practical advantage over simple Full-FT when evaluated rigorously.
• Realistic Standards: FEWTRANS provides a "ruler" for the community to distinguish between methods that are merely overfitting to specific evaluation setups and those that offer genuine robustness.
• Guidance for Future Research:
  • Researchers should focus on pre-training data scale and architecture rather than just transfer algorithms, as these are the dominant factors.
  • Future work should address linguistic domain shifts in multimodal models and develop methods that can handle rare vocabulary without massive fine-tuning.
  • The benchmark encourages the community to move beyond the "validation set illusion" and design algorithms that are robust to hyperparameter uncertainty.

In summary, this work argues that Full-FT is the strongest baseline for few-shot transfer, largely because it leverages the pre-trained model's flat loss landscape through micro-adjustments. The proposed FEWTRANS benchmark and HPE protocol serve to expose the limitations of current evaluation practices and redirect research toward more realistic and robust adaptation strategies.