Technical Summary: Comparing Specialised Small and General Large Language Models on Text Classification

Problem Statement

In Natural Language Processing (NLP), researchers face a critical decision when dealing with limited labelled data: should they utilize a general Large Language Model (LLM) via prompting or in-context learning (without parameter updates), or should they invest a small number of labelled samples to fine-tune or instruction-tune a specialised, smaller model?

While existing literature suggests that general LLMs excel in zero-shot or few-shot settings, and specialised models can outperform them with sufficient data, there is a lack of comprehensive, fair comparisons that determine the precise "break-even point." Previous studies often suffer from methodological inconsistencies, such as comparing specialised models trained on full datasets against general models using only a few samples, or vice versa. Furthermore, many studies ignore the significant performance variance introduced by randomness (e.g., random seeds, data splits) and fail to account for computational efficiency or the impact of model quantisation.

This work addresses the question: "How many labelled samples are required for specialised small models to outperform general large models, while taking performance variance into consideration?"

Methodology

The authors conducted a comprehensive investigation across 8 language models and 8 representative text classification datasets with varying characteristics (binary vs. multi-class, sentence length, task type).

Experimental Design

1. Data-Efficiency Analysis: The study systematically increased the number of labelled training samples from a low number (starting at 1 per class) up to the full dataset size. This allowed for the identification of performance break-even points where specialised models begin to outperform general ones.
2. Approaches Compared:
   • Specialised Models: Fine-tuning (using BERT and RoBERTa) and Instruction-tuning (using Flan-T5, Mistral-7B, and Zephyr-7B).
   • General Models: Zero-shot prompting and Few-shot in-context learning (using Flan-T5, GPT-4, LLaMA-2, LLaMA-3, Mistral-7B, and Zephyr-7B).
3. Variance and Sensitivity: To account for the sensitivity to randomness, experiments were repeated 100 times for fine-tuning and 20 times for other approaches. The study identified two types of break-even points:
   • First Break-Even Point: The point where the average performance of specialised models exceeds general models.
   • Second Break-Even Point: The point where specialised models outperform general models even when accounting for variance (comparing the "worst-case" performance of the better model against the "best-case" of the worse model).
4. Compute-Efficiency and Quantisation: The study compared models of different sizes while ensuring comparable computational costs. It also evaluated the impact of 4-bit quantisation on both performance and variance.
5. Datasets: The evaluation covered 4 binary datasets (SST2, MRPC, CoLA, BoolQ) and 4 multi-class datasets (AG News, TREC, SNIPS, DB Pedia).

Key Contributions and Findings

1. Break-Even Points and Sample Requirements

The study identifies that specialised models often require a surprisingly small number of labelled samples to match or exceed general LLMs:

• Average Requirement: On average, specialised models need only ~100 labelled samples to achieve performance on par with or better than general models in zero-shot or few-shot settings.
• Task Dependency: The required number of samples varies significantly based on dataset characteristics.
  • Multi-class datasets: Fine-tuning often requires only 100–200 samples to outperform general models.
  • Binary datasets: These require significantly more samples (up to 4,000 to outperform zero-shot, and up to 20,000 to outperform instruction-tuned models).
  • Task Complexity: Tasks requiring deeper language understanding (e.g., question answering) or those with longer inputs generally demand more labelled data.

2. Impact of Performance Variance

A major contribution of this work is quantifying the effect of performance variance on the comparison:

• Significant Increase in Requirements: When performance variance is considered (moving from the first to the second break-even point), the number of required labelled samples increases by an average of 100–200%.
• Extreme Cases: In specific scenarios, particularly with in-context learning on certain datasets, the required sample size can increase by 500–3000%, or the break-even point may not be reached even with the full dataset.
• Source of Variance: In-context learning and fine-tuning on very few samples exhibit the highest variance. Fine-tuning variance decreases as the number of labelled samples increases.

3. Model Size and Quantisation

• Size vs. Performance: Larger general models do not consistently lead to better performance or lower variance. In many cases, smaller models (e.g., Flan-T5) outperform larger counterparts (e.g., LLaMA-3, GPT-4) in prompting and in-context learning, particularly on multi-class datasets.
• Quantisation: The use of 4-bit quantisation has a negligible impact on overall performance and variance. In some cases (specifically in-context learning), quantised models even outperform their full-precision counterparts. This suggests quantised models are a highly efficient choice for general LLMs.

4. Instruction-Tuning Efficiency

Instruction-tuned models demonstrated a strong balance between generality and data efficiency. They often achieved near-optimal performance with a fraction of the samples required by standard fine-tuning, making them a highly data-efficient specialised approach.

Significance and Recommendations

The paper concludes that the choice between specialised small models and general large models is not binary but depends heavily on the annotation budget, computational resources, and task characteristics.

• For Limited Annotation Budgets: General LLMs via prompting are preferable for quick prototyping or tasks where classes are not well-defined (e.g., generation tasks).
• For Moderate Annotation Budgets (~100 samples): Specialised models (via fine-tuning or instruction-tuning) are strong contenders and often outperform general models, especially on multi-class tasks.
• For High Variance Sensitivity: Researchers must account for performance variance. Relying on a single run can lead to incorrect conclusions about model superiority. Multiple runs are essential, particularly for in-context learning and low-sample fine-tuning.
• Practical Recommendations:
  • Instruction-tuning is often the optimal solution for large computation budgets, offering consistent benefits across various annotation sizes.
  • 4-bit quantisation should be preferred for general LLMs to improve compute-efficiency with minimal performance loss.
  • Prompt Engineering: The quality and format of prompts and in-context examples are critical factors influencing both performance and variance.

The authors emphasize that while general LLMs show benefits in specific scenarios (e.g., extremely limited annotation or generation tasks), specialised models are highly competitive on text classification tasks with surprisingly few labelled samples, provided that performance variance is properly managed.