Technical Summary: Capturing LLM Capabilities via Evidence-Calibrated Query Clustering

Problem Statement

Large Language Models (LLMs) exhibit diverse capabilities across different domains and task types. Accurate assessment of these strengths is crucial for reliable deployment in personalized settings. However, existing methods for evaluating LLMs often rely on coarse-grained datasets or query groups that yield noisy evidence, failing to characterize latent capability requirements effectively.

Current query clustering approaches typically fall into two categories:

1. Static Human-Labeled Taxonomies: Methods using predefined categories (e.g., "Mathematics," "Chemistry") often fail because a single label can encompass queries requiring vastly different underlying capabilities (e.g., rote recall vs. multi-step deduction).
2. Unsupervised Semantic Embeddings: Clustering based solely on surface-level semantic embeddings often misaligns with actual model performance. Queries with similar semantic topics may require different capabilities, while queries with different topics may share the same latent capability demands.

This misalignment between surface-level semantics and latent capability requirements limits the generalizability of model capability estimates, particularly for unseen queries. Furthermore, existing "LLM-as-a-judge" approaches that extract skill labels often result in hard, discrete partitions that cannot accommodate queries with mixed capability demands and incur high computational costs.

Methodology: Evidence-Calibrated Clustering (ECC)

The authors propose ECC, a novel algorithm that bridges the gap between surface-level semantics and latent capability requirements by integrating prior semantic embeddings with limited posterior model comparisons.

Core Components

1. System Parameters ($\Theta$): ECC characterizes each cluster $k$ using two components:
   • Semantic Centroid ($c_k$): Captures prior semantic signals in the embedding space.
   • Capability Profile ($\theta^{(k)}$): A vector of latent strengths for $M$ LLMs on queries associated with cluster $k$, parameterized by a Bradley-Terry (BT) model.
2. Soft Responsibilities ($P$): To accommodate queries requiring mixed capabilities, ECC introduces trainable soft assignments $r_{qk}$. These weights measure how strongly a query $q$ aligns with the capability profile of cluster $k$, allowing a single query to contribute to multiple clusters.

Optimization Procedure

ECC employs an alternating optimization procedure to jointly learn the system parameters and soft responsibilities:

• Objective Function: The algorithm minimizes an entropy-regularized expected loss combining:
  • Capability Loss ($\ell_{comp}$): Measures how well the cluster's BT profile fits the observed pairwise model comparisons for the query.
  • Embedding Loss ($\ell_{emb}$): Measures the semantic mismatch between the query embedding and the cluster centroid.
  • Entropy Regularization: Encourages diverse assignments and prevents degenerate solutions.
• Alternating Updates:
  1. Update $\Theta$: Given fixed responsibilities, update centroids (as normalized weighted means of embeddings) and capability profiles (by minimizing weighted BT loss).
  2. Update $P$: Given fixed parameters, update responsibilities using a closed-form Boltzmann distribution derived from the combined loss and temperature parameter $T$.

Inference: Probe-Informed Mixture Inference

For unseen queries, ECC employs a lightweight inference strategy:

1. Probe Signal: A single pairwise model comparison is collected for the new query.
2. Calibration: This probe is used to calibrate the query's soft responsibilities ($r_q$) across clusters, reconciling the embedding signal with the posterior evidence.
3. Ranking: The final capability-aware ranking is derived as a weighted mixture of the cluster-specific BT profiles:
   $$P(m_j \succ m_i | q) = \sum_{k=1}^K r_{qk} \sigma(\theta^{(k)}_j - \theta^{(k)}_i)$$

Key Contributions

1. Novel Algorithm: ECC integrates posterior model comparisons with prior embeddings to overcome the misalignment between surface semantics and latent capability demands.
2. Sample-Efficient Estimation: By characterizing clusters with Bradley-Terry capability profiles, ECC enables the estimation of relative model strengths from limited pairwise comparisons.
3. Flexible Structure: The introduction of trainable soft responsibilities allows the model to capture mixed capability requirements, yielding a flexible, capability-aware clustering structure.
4. Empirical Validation: Extensive evaluations demonstrate that ECC produces more capability-coherent clusters and significantly improves ranking quality compared to baselines.

Experimental Results

The authors evaluated ECC on three benchmarks: SPROUT, RouterBench, and Open LLM Leaderboard v2, covering datasets like MMLU, MMLU-Pro, and MATH.

• Ranking Quality: ECC significantly outperformed human-labeled clustering and embedding-only clustering.
  • On unseen queries, ECC improved ranking quality by an average of 17.64 percentage points (pp) over human-labeled clustering and 18.02 pp over embedding-only clustering.
  • ECC consistently achieved the best performance across different inference signals (embedding-only, probe-only, and combined).
• Cluster Coherence: Qualitative analysis showed that ECC clusters are more internally consistent (higher within-cluster BT profile consistency) and better separated (lower inter-cluster similarity) than baselines.
• Downstream Tasks:
  • Query Routing: ECC improved the response quality of routed queries by 16.6% on average over embedding-only clustering.
  • Sample-Efficient Ranking: In tasks involving ranking new models with limited comparison budgets, ECC achieved 16.3 pp higher ranking quality than embedding-only baselines.
• Robustness: ECC demonstrated robustness to variations in the number of clusters, probe selection, embedding models, and tied comparisons.

Significance and Claims

The paper claims that ECC addresses a fundamental limitation in current LLM evaluation: the reliance on surface-level semantics that do not reflect true capability distributions. By calibrating semantic priors with posterior evidence, ECC provides a more accurate, interpretable, and flexible framework for capability-aware evaluation.

The authors emphasize that ECC is training-free (requiring no gradient-based optimization of neural components) and computationally efficient, making it suitable for resource-constrained environments. The method enables fine-grained adaptation across heterogeneous queries and supports downstream applications like optimal query routing and efficient new model ranking without the need for exhaustive benchmarking.

The paper concludes that while it currently focuses on text-only queries, the framework is modality-agnostic and could theoretically accommodate multimodal inputs. It also notes that future work could extend the method to context-aware settings involving conversational history.