Iterative Definition Refinement for Zero-Shot Classification via LLM-Based Semantic Prototype Optimization

1. Problem Statement

Web content classification is essential for cybersecurity (e.g., blocking malicious URLs) and compliance. However, traditional supervised methods require large annotated datasets and struggle with the dynamic nature of the web. Zero-shot classification using embedding models offers a solution by mapping text and category descriptions into a shared semantic space without task-specific training.

The Core Limitation: The performance of embedding-based zero-shot classification is fundamentally tied to the quality of the category definitions (semantic prototypes).

• If definitions are vague, overly general, or semantically similar across classes, the resulting prototypes in the embedding space overlap.
• This leads to systematic misclassification, even when using state-of-the-art embedding models.
• Current approaches either fine-tune models (expensive, requires data) or use static definitions (brittle). There is no mechanism to iteratively improve the definitions themselves without retraining the model.

2. Methodology

The authors propose a training-free, adaptive framework that uses Large Language Models (LLMs) to iteratively refine category definitions based on classification feedback. The embedding model remains frozen throughout the process.

2.1. Core Workflow

1. Initialization: Category definitions are initialized (either minimally or via an LLM).
2. Embedding & Classification: Text and definitions are embedded using a frozen foundation model. Labels are assigned via cosine similarity.
3. Feedback Loop: The system calculates performance metrics (Macro F1) and identifies failure modes (e.g., misclassified instances).
4. Refinement: An LLM receives the current definitions and specific feedback signals to generate improved definitions.
5. Iteration: The process repeats, updating definitions until convergence.

2.2. Three Refinement Strategies

The paper introduces three distinct strategies to guide the LLM's optimization:

• M1: Example-Guided Refinement

  • Input: Current definitions + a random labeled sample + current Macro F1 score.
  • Goal: Ground definitions in real data characteristics to ensure they capture the semantic essence of the category.

• M2: Confusion-Aware Refinement

  • Input: Current definitions + samples from the top-$k$ most confused class pairs (derived from the confusion matrix) + current Macro F1.
  • Goal: Specifically target decision boundaries where the classifier fails, forcing the LLM to make confusing categories more distinguishable.

• M3: History-Aware Refinement with Stochastic Acceptance

  • Input: Current definitions + the history of the last $m$ definition sets and their scores + confusion samples.
  • Mechanism: To prevent the LLM from getting trapped in local optima, the authors implement a Monte Carlo acceptance rule (inspired by Simulated Annealing).
  • Logic: If a new definition improves performance, it is accepted. If it degrades performance, it is accepted with a probability $P = \exp(-\Delta E / T_t)$, where $T_t$ is a temperature parameter that decreases over time. This allows for "exploration" of the definition space early on and "exploitation" (greedy acceptance) later.

3. Key Contributions

1. Novel Framework: The first framework to apply iterative semantic prototype refinement to web content classification without updating model parameters. It shifts the optimization focus from model weights to prompt/definition quality.
2. Strategic Innovation: Introduction of three refinement strategies (Example, Confusion, History) and the integration of a Monte Carlo acceptance rule to navigate the definition search space effectively.
3. Comprehensive Benchmarking:
   • Evaluated across 13 state-of-the-art embedding foundation models (ranging from 278M to 1B parameters).
   • Tested with 4 different LLMs (Gemma, Mistral, GLM, Qwen) for the refinement loop.
   • Demonstrated that specific LLM/Embedding pairings yield optimal results, providing deployment guidance.
4. New Dataset (B2MWT-10C): Release of a human-labeled benchmark containing 10,000 URLs across 10 semantically overlapping categories (e.g., "Betting & Gambling" vs. "Leisure & Entertainment"), designed specifically to test the limits of zero-shot classification.

4. Experimental Results

The framework was tested on the N24News dataset and the new B2MWT-10C dataset.

• Performance Gains:
  • Consistent Improvement: Every embedding model showed performance gains after refinement. No refinement run degraded performance relative to the baseline.
  • Magnitude: On B2MWT-10C, the best model (Voyage-4-nano) improved from a baseline of 62.65 to 75.49 (+12.84 points).
  • High Impact on Weak Baselines: Models with initially poor performance saw the largest relative gains. For example, Multilingual-e5-large on N24News improved from 49.30 to 69.20 (+19.90 points).
• Strategy Comparison:
  • M3 (History-Aware) generally outperformed M1 and M2, particularly when combined with the stochastic acceptance rule.
  • Confusion-Aware (M2) provided significant gains over Example-Guided (M1), proving that targeting specific failure points is more effective than random sampling.
• Visual Evidence:
  • t-SNE Plots: Showed that initially overlapping category prototypes (e.g., "Media" vs. "Opinion") became well-separated after refinement.
  • Confusion Matrices: Demonstrated a shift toward diagonal dominance, indicating reduced confusion between semantically similar classes.

5. Significance and Conclusion

This paper establishes definition quality as a critical, yet underexplored, factor in embedding-based systems.

• Cost-Efficiency: It offers a "training-free" alternative to fine-tuning. Organizations can improve classification accuracy by simply optimizing their text prompts/definitions using an LLM, avoiding the computational cost and data requirements of retraining embedding models.
• Adaptability: The framework is highly adaptable to new categories or evolving web content, as definitions can be refined on-the-fly based on real-world misclassification feedback.
• Future Directions: The authors suggest extending this approach to visual modalities and Vision-Language Models (VLMs), indicating a broader applicability beyond text-only classification.

In summary, the paper demonstrates that iteratively refining the "questions" (definitions) is just as powerful as refining the "answers" (model weights) for zero-shot classification tasks.