🤖 machine learning

Automatic Combination of Sample Selection Strategies for Few-Shot Learning

This paper introduces ACSESS, a method that automatically combines various sample selection strategies to consistently outperform individual strategies and match or exceed specialized baselines across diverse few-shot learning models and datasets, demonstrating particular effectiveness when selecting very few shots.

Original authors: Branislav Pecher, Ivan Srba, Maria Bielikova, Joaquin Vanschoren

Published 2026-04-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Branislav Pecher, Ivan Srba, Maria Bielikova, Joaquin Vanschoren

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to teach a very smart, but inexperienced, sous-chef how to cook a new dish. You only have a tiny amount of time and a very small basket of ingredients to show them. This is the world of Few-Shot Learning: teaching an AI to do a new task with very few examples.

The big question the paper asks is: Which ingredients should you pick from your giant pantry to show the sous-chef?

If you pick random ingredients, the chef might get confused. If you pick the "best" ingredients, the chef learns faster. But what makes an ingredient "best"? Is it the most colorful one? The one that tastes the most different? The one that is hardest to chop?

The Problem: Too Many Rules, One Size Doesn't Fit All

For a long time, researchers have had different "rules of thumb" for picking these ingredients (samples):

The Similarity Rule: "Pick ingredients that look exactly like what we are trying to cook."
The Diversity Rule: "Pick ingredients that are all very different from each other to show variety."
The Difficulty Rule: "Pick the ingredients that are the hardest to prepare, because that's where the real learning happens."

The problem is that in the era of giant AI models (Large Language Models), researchers started inventing brand new, complicated rules just for those specific models. They forgot about the old, reliable rules that worked well for regular computers. Also, nobody knew which rule was actually the best, or if mixing them together would work better.

The Solution: ACSESS (The Smart Sous-Chef Manager)

The authors of this paper created a new method called ACSESS (Automatic Combination of SamplE Selection Strategies).

Think of ACSESS as a smart manager who doesn't just follow one rule. Instead, the manager looks at all the different rules (Similarity, Difficulty, Diversity, etc.) and asks: "Which combination of these rules will give us the best results for this specific job?"

Here is how the manager works:

The Taste Test: The manager tries out different combinations of rules on a small test kitchen.
The Scorecard: It sees which combination helps the AI learn the fastest.
The Final Mix: It creates a custom "recipe" for picking ingredients. Sometimes it needs 70% "Difficulty" and 30% "Diversity." Other times, it might need 50% "Similarity" and 50% "Easy-to-learn."

The Big Discoveries

The researchers tested this manager on 14 different datasets (like news articles, voice commands, and pictures of animals) and found some surprising things:

1. The "Hard" Lessons are Gold (for some chefs)
For the giant AI models (the "In-Context Learning" chefs), the best ingredients to show them were the hardest ones to learn. It's like showing a student the most complex math problem first; if they can solve that, the easy ones are a breeze. The AI learns best from the "toughest" examples.

2. The "Easy" Lessons are Gold (for others)
For the traditional AI models (the "Gradient" chefs), the best ingredients were the easy ones or the ones that were "forgotten" the least. These models prefer a steady, clear foundation.

3. Mixing is Better than Picking One
The most important finding is that no single rule is perfect. The manager (ACSESS) that mixed the rules together always beat the chefs who followed just one rule. It's like saying, "Don't just pick the spiciest pepper; pick a mix of spicy, sweet, and crunchy to make the perfect sauce."

4. Less is More (But only up to a point)
The researchers found that picking the right 5 ingredients is way better than picking 50 random ones.

The Sweet Spot: When you have very few examples (1 to 5), picking the right ones makes a huge difference.
The Tipping Point: Once you have about 30 to 40 examples, it doesn't matter much which ones you pick anymore. The AI learns well enough that picking the "perfect" ones stops helping. In fact, for the giant AI models, adding too many examples (like 50) actually made them perform worse because their "memory" (context window) got too full and confused.

The Takeaway

This paper is like a guide for a teacher. It tells us:

Don't just guess which examples to show your AI.
Don't rely on just one "trick" to pick them.
Instead, use a smart system (ACSESS) that mixes different strategies to find the perfect, small handful of examples.
And remember: Quality over quantity. A few perfectly chosen examples are often better than a mountain of random ones.

By using this method, we can make AI smarter, faster, and more efficient, even when we don't have a lot of data to teach it with.

1. Problem Statement

Few-shot learning (FSL) aims to adapt models to new tasks with only a handful of labeled samples. The performance of FSL is highly sensitive to the quality and selection of these few samples (the "shots").

The Gap: While effective sample selection strategies exist for supervised settings (e.g., active learning, core-set selection), research on Large Language Models (LLMs) and in-context learning (ICL) largely overlooks them. Instead, recent works propose strategies tailored specifically to individual ICL settings, often ignoring well-established, general-purpose objectives.
The Challenge: Single-objective strategies (e.g., selecting only for similarity or only for diversity) often yield suboptimal results because they fail to capture the complementary properties of samples (e.g., a sample might be highly informative but hard to learn). Furthermore, existing ICL-specific baselines do not generalize well across different tasks and models.

2. Methodology: ACSESS

The authors propose ACSESS (Automatic Combination of SamplE Selection Strategies), a method that automatically identifies and combines multiple sample selection objectives to leverage their complementarity.

Core Components

The method operates in three stages:

Definition of Single-Property Strategies: The authors curate a pool of 23 established strategies categorized by three properties:
- Informativeness: Similarity, Diversity, Active Learning (Entropy, Margin, Least Confidence, Loss), and Core-Set methods (CAL, DeepFool, GraNd, Graph-Cut).
- Representativeness: Herding, KCenter, CRAIG, Glister.
- Learnability: Forgetting (how often a sample is misclassified after being correct) and Cartography (categorizing samples as easy, hard, or ambiguous based on training dynamics).
Identification of Relevant Strategies: Instead of brute-forcing all combinations (which is computationally infeasible), ACSESS uses three distinct search methods to find a subset of high-performing strategies ( $S_{final}$ ):
- Forward Selection: Iteratively adds the strategy that yields the highest performance increase.
- Backward Selection: Iteratively removes the strategy whose removal causes the least performance drop (or highest gain).
- Datamodels Selection: Adapts the "Datamodels" approach (Ilyas et al., 2022). It trains a LASSO linear regression model where features are the presence/absence of strategies in random combinations, and the target is the performance gain over a baseline. This identifies strategies with positive weights.
- Intersection: The final set of strategies is the intersection of the sets identified by all three methods ( $S_{final} = S_F \cap S_B \cap S_D$ ), ensuring robustness and efficiency.
Automatic Combination & Scoring:
- Each sample $x$ is assigned a score based on a weighted sum of the objectives from the selected strategies:
  $\text{score}(x) = \sum_{s \in S_{final}} w_s \times \text{objectives}_s(x)$
- Weighting Schemes:
  - Uniform: Equal weights ( $1/|S_{final}|$ ). Low cost, high transferability.
  - Weighted: Uses the weights learned by the Datamodels LASSO. High performance, dataset/model-specific.
  - Weighted + Random: Includes random selection as a baseline strategy within the weighting scheme.

3. Experimental Setup

Scope: A large-scale study involving 23 selection strategies, 5 in-context learning models (Mistral-7B, Zephyr-7B, LLaMA-3.1-8B, Gemma-3-4B, Qwen-2.5-7B), and 3 gradient-based FSL approaches (Prototypical Networks, MAML, Few-Shot Fine-Tuning).
Datasets: 6 text datasets (news classification, intent classification) and 8 image datasets (Meta-Album).
Baselines: Compared against Classic (random per task), Random (fixed set), and state-of-the-art ICL-specific strategies (LENS, Active Prompt, EXPLORA, CASE).

4. Key Results

Superior Performance: ACSESS consistently outperforms all individual single-property strategies.
- ICL: Achieved up to 2.5% accuracy improvement over the best single strategy and outperformed ICL-specific baselines (e.g., +0.95% over LENS, +0.16% over CASE).
- Gradient FSL: Achieved up to 1.8% improvement.
Complementarity: The method successfully identified that learnability (specifically "hard to learn" samples for ICL and "easy/ambiguous" for gradient FSL) combined with decision boundary properties (Margin) and forgetting metrics provides the most benefit. Representativeness strategies were largely excluded by the automatic selection.
Shot Count Sensitivity:
- Sample selection provides the greatest benefit at low shot counts (1–5 shots).
- As the number of shots increases (30–40+), the benefit of sophisticated selection diminishes, and strategies regress to the performance of random selection.
- For ICL, increasing shots beyond ~25–50 often leads to performance degradation due to context window limits.
Dataset Size: Sample selection remains effective even when the available dataset is small (e.g., selecting from only 10–25% of the full dataset yields performance comparable to using the full dataset).

5. Key Contributions

First Large-Scale Study: The first comprehensive evaluation of 23 diverse sample selection strategies across 5 LLMs, 3 FSL approaches, and 14 datasets.
ACSESS Framework: A novel, automated method to combine selection objectives, proving that well-established strategies (like active learning and core-sets) can rival or exceed strategies specifically designed for ICL.
Insight on Learnability: The discovery that learnability is a stronger indicator of sample quality for FSL than informativeness or representativeness, a factor often overlooked in prior ICL research.
Practical Guidelines: Demonstrated that while full ACSESS (weighted, dataset-specific) is optimal, a uniform combination of a few key strategies (Cartography + Margin + Forgetting) offers a computationally cheap, transferable alternative with near-optimal performance.

6. Significance

This work challenges the notion that few-shot learning requires bespoke, task-specific sample selection heuristics. By demonstrating that combining established, general-purpose objectives yields superior and more robust results, ACSESS provides a scalable, data-centric approach to improving FSL. It highlights that for low-resource settings, curating a high-quality subset of samples is more critical than the sheer volume of data, and that the "best" strategy depends heavily on the learning paradigm (ICL vs. Gradient) and the specific properties of the data (noise, class imbalance).