BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMs

This paper introduces BTZSC, a comprehensive benchmark of 22 datasets designed to systematically evaluate and compare the zero-shot text classification capabilities of NLI cross-encoders, embedding models, rerankers, and instruction-tuned LLMs, revealing that modern rerankers currently achieve state-of-the-art performance while embedding models offer the best accuracy-latency trade-off.

The Gist

Imagine you are the manager of a massive library. Your job is to sort thousands of new books into the right shelves (like "Mystery," "Cooking," or "Science Fiction") without ever having seen those specific books before. You don't have time to read every single page to learn the rules. Instead, you have to look at the title and the back-cover blurb and guess where it belongs based on your general knowledge of the world.

This is exactly what Zero-Shot Text Classification does for computers. It's the ability for an AI to sort text into categories it has never been explicitly taught, just by understanding the meaning of the words.

For a long time, there was a debate in the AI world: Which tool is the best for this job?

1. The "Logic Puzzlers" (NLI Cross-Encoders): These models were trained on logic games (like "If A is true, is B true?"). They are smart but can be slow and rigid.
2. The "Indexers" (Embedding Models): These turn text into a list of numbers (a vector) and find matches by how close the numbers are. They are fast but sometimes miss the nuance.
3. The "Refiners" (Rerankers): These are like a second pair of eyes that double-check the Indexer's work. They look at the text and the category description together to make a final, precise decision.
4. The "Geniuses" (LLMs): These are the big, powerful chatbots (like the one you are talking to now) that can write stories and answer questions. They can also sort text, but they are heavy, slow, and expensive to run.

The Problem: A Messy Race Track

Until now, comparing these four tools was like trying to race a Formula 1 car, a bicycle, a skateboard, and a horse on different tracks. Some benchmarks (tests) only looked at the bicycle, others only at the horse, and some tests cheated by letting the racers practice on the exact track they were about to run on. We didn't have a fair, unified race to see who was actually the fastest and most accurate.

The Solution: BTZSC (The Great AI Sorting Olympics)

The authors of this paper created BTZSC (Benchmark for Textual Zero-Shot Classification). Think of this as building a massive, standardized obstacle course with 22 different challenges.

• The Course: It includes tasks like sorting movie reviews (Sentiment), figuring out what a customer wants (Intent), categorizing news articles (Topic), and detecting emotions in chat logs (Emotion).
• The Racers: They tested 38 different AI models from all four families mentioned above.
• The Rules: No cheating! The models had to guess the categories using only their pre-existing knowledge, with no extra training on the test data.

The Results: Who Won the Race?

Here is what the "Great Sorting Olympics" revealed, using some simple analogies:

1. The New Champions: Rerankers (The "Specialized Editors")

• Winner: The Qwen3-Reranker-8B took the gold medal with a score of 0.72 (on a scale where 1.0 is perfect).
• Analogy: Imagine a librarian who first quickly glances at a book (Indexer) and then pulls it off the shelf to read the first paragraph carefully before deciding where it goes. This "double-check" method is incredibly effective. These models are the new state-of-the-art, beating everything else.

2. The Value Kings: Embedding Models (The "Speedy Librarians")

• Performance: Models like GTE-large came in a very close second.
• Analogy: These are like librarians who have memorized the Dewey Decimal System perfectly. They don't read the whole book; they just look at the ISBN and instantly know the shelf. They aren't quite as accurate as the "Specialized Editors," but they are much faster and cheaper to run. If you need to sort millions of books in a second, these are your best bet.

3. The Heavy Hitters: Instruction-Tuned LLMs (The "Genius Scholars")

• Performance: Big models like Mistral and Qwen did very well, especially on complex topics, but they didn't beat the Rerankers.
• Analogy: These are like PhD students who read the whole book, analyze the author's tone, and write a 5-page essay before picking a shelf. They are brilliant and understand nuance, but they are slow and expensive. For simple sorting tasks, they are often "overkill" and get outperformed by the specialized tools.

4. The Old Guard: NLI Cross-Encoders (The "Logic Puzzlers")

• Performance: They are still good, but they hit a "ceiling." Making them bigger didn't make them much better.
• Analogy: They are like a very smart person trying to solve a puzzle using only logic rules. They work well, but they've reached the limit of how much they can improve just by getting bigger.

The Big Takeaways

• Size isn't everything: A massive 12-billion-parameter "Genius" model didn't beat a smaller, specialized 8-billion-parameter "Editor" (Reranker). Specialization wins.
• Speed vs. Smarts: If you need the absolute best accuracy, use a Reranker. If you need to sort text instantly on a phone or a cheap server, use a modern Embedding Model.
• The "Zero-Shot" Reality: We finally know that these models can actually understand human language descriptions of categories without needing to be retrained for every single new job.

Why This Matters

Before this paper, researchers were shouting past each other, each claiming their specific tool was the best. BTZSC is the referee that finally settled the score. It provides a fair playing field so that in the future, we can build better, faster, and smarter AI tools for sorting everything from spam emails to mental health support chats, without needing to hire armies of human annotators to teach the computer every single rule.

In short: We now have a clear map of the AI landscape. We know which tool to grab for the job, saving us time, money, and frustration.

Technical Summary

Here is a detailed technical summary of the paper "BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMs".

1. Problem Statement

Zero-Shot Text Classification (ZSC) aims to assign labels to text without task-specific training, relying instead on semantic matching between input text and human-readable label descriptions. While ZSC eliminates the cost of data annotation, the field lacks a unified, fair evaluation framework.

• Limitations of Existing Benchmarks: Current benchmarks like MTEB (Massive Text Embedding Benchmark) primarily evaluate embedding models using supervised linear probes (training a classifier on top of frozen embeddings), which fails to test the genuine zero-shot capabilities of the models.
• Fragmented Evaluation: Previous studies often focus on a single model family (e.g., only NLI cross-encoders or only LLMs) or a narrow set of datasets, making it impossible to compare the relative strengths of diverse architectures (NLI cross-encoders, embeddings, rerankers, and LLMs) under consistent zero-shot conditions.

2. Methodology: The BTZSC Benchmark

The authors introduce BTZSC, a comprehensive benchmark designed to systematically evaluate four major model families under a strict zero-shot protocol.

A. Dataset Construction

• Scope: 22 English single-label datasets spanning four core classification categories: Sentiment, Topic, Intent, and Emotion.
• Diversity: The datasets cover diverse domains (news, social media, finance, politics, dialogue), varying class cardinalities (from binary to 77 classes), and document lengths (micro-texts to long articles).
• Preprocessing: All datasets are standardized into a single text input and a categorical label. Crucially, no labeled examples from these datasets are used for training, hyperparameter tuning, or model selection.

B. Evaluation Protocol

• Label Verbalization: Labels are converted into natural language descriptions (e.g., "The sentiment is {label}").
• Inference Strategies:
  • NLI Cross-Encoders: Treat the text as a premise and the label description as a hypothesis; score entailment.
  • Embedding Models: Encode text and label descriptions separately; select the label with the highest cosine similarity.
  • Rerankers: Treat text as a query and label descriptions as documents; re-rank based on relevance scores.
  • Instruction-tuned LLMs: Frame the task as a multiple-choice problem, prompting the model to select the best option based on next-token probability.
• Primary Metric: Macro F1 (to handle class imbalance and varying cardinalities), supplemented by Micro Accuracy, Precision, and Recall.

C. Model Families Evaluated

The study evaluates 38 models across four categories:

1. Base Transformer Encoders: Unfined-tuned models (e.g., BERT, DeBERTa-v3, ModernBERT).
2. NLI Cross-Encoders: Models fine-tuned on Natural Language Inference datasets (e.g., BART-large-MNLI, DeBERTa-v3-NLI).
3. Embedding Models: Sentence encoders (e.g., all-MiniLM-L6-v2, E5, BGE, GTE, Qwen3-Embedding).
4. Rerankers: Models designed for re-ranking query-document pairs (e.g., MS MARCO MiniLM, BGE-Reranker, Qwen3-Reranker).
5. Instruction-tuned LLMs: Generative models (e.g., Gemma, Llama, Phi, Qwen, Mistral) ranging from 270M to 12B parameters.

3. Key Results

The experiments reveal distinct performance regimes and scaling behaviors:

• State-of-the-Art Performance:

  • Modern Rerankers achieve the highest performance. Qwen3-Reranker-8B sets a new SOTA with a macro F1 of 0.72, significantly outperforming all other families.
  • Even the smaller Qwen3-Reranker-0.6B outperforms all NLI cross-encoders, demonstrating the strength of the reranker architecture even at moderate scales.

• Embedding Models:

  • Strong embedding models (e.g., GTE-large-en-v1.5, E5-large-v2) achieve macro F1 scores around 0.60–0.62.
  • They offer the best trade-off between accuracy and latency, making them highly suitable for practical deployment.
  • Unlike rerankers, scaling embedding models (e.g., from 0.6B to 8B parameters) yields diminishing returns, with performance plateauing around 0.60–0.62 F1.

• Instruction-tuned LLMs:

  • Small LLMs (<1B) perform poorly, comparable to base encoders.
  • Mid-to-large LLMs (4B–12B, e.g., Qwen3-8B, Mistral-Nemo-12B) achieve competitive results (macro F1 ~0.66–0.67), particularly excelling in Topic Classification.
  • However, they still trail specialized rerankers by ~5 F1 points while incurring significantly higher computational costs and latency.

• NLI Cross-Encoders:

  • These models show clear improvements over base encoders but have plateaued in recent years.
  • Scaling NLI models (Base vs. Large) provides modest gains (+3.5 F1 points), but they cannot match the performance of modern rerankers or strong embeddings.

• Scaling Trends:

  • Rerankers show monotonic performance gains with scale.
  • LLMs show the steepest scaling curve, improving rapidly between 3B and 8B parameters.
  • Embeddings saturate quickly after a few hundred million parameters.

• Task Difficulty:

  • Sentiment is the easiest task (Median F1 ≈ 0.88–0.9).
  • Topic and Intent are intermediate.
  • Emotion is the most challenging (Median F1 ≈ 0.25–0.35) for all model families.

4. Key Contributions

1. Unified Benchmark (BTZSC): The first benchmark to jointly evaluate NLI cross-encoders, embeddings, rerankers, and LLMs under a consistent, strict zero-shot protocol.
2. Comprehensive Dataset: A curated set of 22 diverse datasets covering sentiment, topic, intent, and emotion, filling gaps left by previous benchmarks.
3. Systematic Comparison: A rigorous analysis of 38 models, revealing that rerankers currently lead in accuracy, while embeddings lead in efficiency.
4. Insight on Scaling: Demonstrates that scaling benefits rerankers and LLMs significantly more than embedding models, which tend to saturate earlier.
5. Reproducibility: Full release of code, model checkpoints, and a live leaderboard to ensure fair and reproducible progress.

5. Significance and Implications

• Shift in Paradigm: The paper challenges the dominance of NLI-based cross-encoders, showing that rerankers (often derived from generative architectures) are now the superior choice for high-accuracy zero-shot classification.
• Deployment Guidance: For real-world applications requiring low latency, strong embedding models (like GTE or E5) offer the optimal balance of performance and speed, outperforming NLI models and being more efficient than LLMs.
• LLM Viability: While instruction-tuned LLMs are competitive, they are not yet the most efficient solution for ZSC unless the task requires complex reasoning or few-shot capabilities beyond simple classification.
• Future Research: The benchmark highlights the need for better label verbalization techniques and further exploration of scaling laws for rerankers and embeddings in zero-shot scenarios.

In conclusion, BTZSC provides a critical foundation for understanding the current landscape of zero-shot text classification, establishing that while NLI models laid the groundwork, modern rerankers and high-quality embeddings are the current leaders in the field.