Technical Summary: SELECT-LLM

Problem Statement

The rapid proliferation of Large Language Models (LLMs) has created a significant model selection challenge. While modern LLMs possess strong zero-shot capabilities, their relative performance is highly context-dependent, varying substantially across domains, tasks, and languages. Identifying the optimal LLM for a specific application typically requires evaluating candidates against fixed benchmark sets. However, standard benchmarking is increasingly difficult to use for selection because benchmarks often lag behind new model releases and may not reflect domain-specific needs.

A more practical alternative involves using small, annotated subsets of queries to select the best model. However, random or heuristic-based selection of these subsets often leads to inefficient use of annotation budgets and fails to reveal meaningful performance differences among candidates. While active learning has been extensively studied for classification tasks, existing methods for generative settings are limited, often restricting comparisons to two models or focusing on single-model testing rather than selection from arbitrary pools of candidates. The core problem addressed is how to reliably identify the best LLM for a target task and data distribution under limited annotation resources.

Methodology: SELECT-LLM

The authors propose SELECT-LLM, a framework for the active model selection of LLMs. The goal is to select a small set of $b$ queries (where $b \ll n$) from a pool of $n$ unannotated queries such that their annotations maximally reduce uncertainty about the identity of the best model.

Formulation

The problem is cast as maximizing mutual information between the unknown identity of the best model ($F$) and the acquired annotations ($A$). The authors define the best model as the one achieving the highest average performance score over the query set when evaluated against ground-truth references.

To make this tractable, they adopt a probabilistic formulation where the probability of a model output is proportional to its similarity to the reference answer, governed by a temperature parameter $\tau$. The objective is to select queries that maximize the expected information gain:
$$A_{opt}[b] = \arg \max_{A \subseteq \{(q_i, r_i)\}, |A| \le b} I(F; A)$$

Algorithm and Approximation

Direct computation of information gain is intractable for open-ended generation due to the unbounded reference space. SELECT-LLM derives a tractable selection rule through three key approximations:

1. Taylor Approximation: The logarithmic term in the mutual information objective is approximated using a second-order Taylor expansion, converting the objective into a quadratic form involving pairwise comparisons.
2. Linearization: The likelihood model is locally linearized, replacing differences in probability with differences in similarity scores between model outputs and potential references.
3. Similarity Substitution: The average over possible reference responses is approximated by the dissimilarity between the candidate model outputs themselves.

This results in a greedy selection rule (Equation 5) that selects the next query $q_t$ by minimizing the posterior-weighted pairwise similarity between candidate model outputs:
$$q_t = \arg \min_{q \in U_t} \sum_{j=1}^{m} \sum_{k=1}^{m} P(F = f_j | A_t) P(F = f_k | A_t) s(f_j(q), f_k(q))$$
Intuitively, the algorithm prioritizes queries where the most plausible candidate models produce the most dissimilar responses, thereby maximizing the information gained from a single annotation.

Model-Agnostic Nature

A critical feature of SELECT-LLM is that it relies only on generated model responses. It does not require access to model weights, logits, or internal states. This makes the framework applicable to both open-weight and black-box (API-only) LLMs.

Parameter Selection

The temperature parameter $\tau$, which controls the sharpness of the posterior distribution, is selected prior to the active selection process without requiring oracle annotations. The authors use a noisy proxy signal derived from pairwise similarities between model responses to perform a grid search and identify the $\tau$ that yields the best identification performance.

Key Contributions

1. First Formulation: To the authors' knowledge, this is the first formulation and study of active model selection for LLMs under limited annotation budgets.
2. SELECT-LLM Framework: A principled, model-agnostic framework that greedily selects queries to maximize information gain. It derives a tractable approximation based on pairwise similarities between model outputs, avoiding the need for internal model access.
3. Comprehensive Evaluation: The framework is validated across 23 diverse datasets (covering reasoning, QA, summarization, translation, etc.) and 156 evaluated models (including open-source and API-only models) using multiple reference-based metrics (BERTScore, ROUGE, BLEU, F1, Exact Match, etc.).

Results

The evaluation demonstrates that SELECT-LLM consistently outperforms adapted baseline strategies (including Random, Margin, Min Agreement, VMA, and AMC) across all settings:

• Best Model Identification: SELECT-LLM identifies the true best model using significantly fewer annotations than the strongest baseline. Annotation cost reductions reached up to 81.8% (e.g., on the XSum dataset with ROUGE-L).
• Near-Best Model Selection: When the goal is relaxed to identifying a model within a small performance gap ($\delta$) of the best, SELECT-LLM achieves even higher efficiency, with annotation reductions up to 84.78%.
• Robustness: In terms of the 95th percentile performance gap (the worst-case difference between the selected model and the true best), SELECT-LLM consistently maintains the lowest gap across datasets and identification probability levels (70%, 80%, 90%, 100%), indicating robustness even in difficult realizations.
• Generalization: The method performs well across diverse task families and evaluation metrics, indicating it is not tied to a specific scoring rule or domain.

Significance and Claims

The paper claims that SELECT-LLM provides a practical approach to active model selection in realistic black-box settings. By relying solely on model outputs and an external oracle, it bridges the gap between theoretical active learning and the practical constraints of modern LLM deployment.

The authors note that while the method significantly reduces annotation costs for model selection, it operates under specific assumptions:

• It assumes a reliable oracle signal (reference-based evaluation) rather than noisy human preferences or LLM-as-a-Judge systems.
• It assumes the candidate pool and query pool are fixed before selection begins.

The work highlights that efficient selection can make rigorous comparison of LLMs more accessible but also warns that biased annotations or evaluation criteria could lead to biased model choices, particularly in safety-critical domains. The authors position this as a foundational step toward more efficient and reliable LLM evaluation protocols.