💻 computer science

When is Routing Meaningful? Diversity and Robustness in Language Model Societies

This paper argues that for routing in language model societies to be meaningful, it must ensure both behavioral diversity among actors and stability against query perturbations, introducing metrics like Hierarchic Social Entropy and perturbation-based robustness to reveal that high accuracy alone does not guarantee effective specialization and that smaller, curated agent subsets can often recover most available diversity.

Original authors: Fantine Huot, Michael Kaisers, Mirella Lapata

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fantine Huot, Michael Kaisers, Mirella Lapata

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a massive, chaotic library of 112 different robots, each with its own personality and skill set. You want to build a "Librarian" (a router) that decides which robot should answer a visitor's question. Usually, people only check if the Librarian gets the right answer or saves money. But this paper argues that's like judging a conductor only by whether the orchestra plays the right notes, ignoring whether the musicians are actually different people or if the conductor gets confused by a slight change in the sheet music.

The authors, Fantine Huot, Michael Kaisers, and Mirella Lapata, suggest that for a routing system to be truly "meaningful," it needs two special ingredients: Diversity and Stability.

The Two Rules of the Game

1. The "Not-All-Alike" Rule (Diversity)
If every robot in your library answers every question exactly the same way, having a Librarian is pointless. It's like hiring a traffic cop to direct cars when all the roads lead to the same dead end. The paper introduces a fancy math tool called Hierarchic Social Entropy (HSE) to measure how different the robots actually are. Think of HSE as a "personality detector." If the score is zero, everyone is a clone, and routing is just a waste of time.

2. The "Don't Get Confused by a Typo" Rule (Stability)
Imagine a visitor asks, "How do I bake a cake?" and the Librarian sends them to Robot A. But if the visitor types, "How do I bake a cakee?" (with an extra 'e'), the Librarian suddenly sends them to Robot B. That's chaos! The paper argues that a good routing policy must be robust. It shouldn't care about tiny typos, weird sentence structures, or adding random words like "The sky is blue" before the question. If the Librarian can't handle these surface-level changes, the robots can never get good at their specific jobs because they never get a steady stream of similar requests.

The Big Surprise: Less is More

The team tested these ideas on two giant datasets: EmbedLLM (with 112 models) and RouterBench (with 11 models). They found something wild: You don't need a huge library to get great diversity.

In these simulations, they discovered that a carefully picked group of fewer than ten agents (specifically, about nine for EmbedLLM and four for RouterBench) captures almost all the unique "personalities" available in the massive pool. It's like finding that a small, perfectly curated band of four musicians can play a wider variety of songs than a hundred random people shouting in a room. The paper suggests this is a practical "coreset" (a small, perfect subset) for designing these societies.

The Accuracy Trap: Being Right vs. Being Stable

Here is where it gets tricky. The paper measured two types of routing policies:

KNN Routers: These are like smart search engines that look at the "shape" of the question in a mathematical space to find the best robot.
Prompted Routers: These are like human-like assistants who read the question and the robot's job description to make a choice.

The results showed a sharp trade-off. The KNN routers were amazing at getting high accuracy on clean questions when they were paired with "specialist" societies (robots designed for specific jobs). However, the moment you added a typo or a weird rephrasing, their performance collapsed. They became unstable, sending the same question to different robots just because the words changed slightly.

In contrast, the Prompted routers were a bit less accurate on the clean questions, but they were rock solid. They stayed stable across all types of typos and rewrites. The paper illustrates that you can have high accuracy without meaningful routing, but if you want a system that actually works in the real world (where people make typos), you need that stability.

What This Means (and What It Doesn't)

The paper suggests that we've been looking at routing the wrong way. We can't just chase the highest accuracy score; we have to check if the society of robots is actually diverse enough to need a router, and if the router is stable enough to handle real-world messiness.

They ruled out the idea that a huge number of models automatically equals a diverse society. Their measurements show that many real-world models are actually quite similar in how they behave, so adding more of them doesn't help much.

They also cautioned that their "specialist" societies were built with perfect, binary rules (a robot either knows a topic or it doesn't), which is a bit more rigid than real life. In the real world, experts might overlap more, which could soften the sharp drops in performance the KNN routers experienced.

So, the next time you see a system routing tasks to different AI models, remember: it's not just about who gets the answer right. It's about whether the team is actually a team of different experts, and whether the manager can keep them on track even when the visitors start stuttering.

Technical Summary: When is Routing Meaningful? Diversity and Robustness in Language Model Societies

Problem Statement

Current evaluation of routing policies for multi-model systems (societies of language models, tools, or agents) focuses almost exclusively on downstream utility: task accuracy and inference cost. The authors argue that high accuracy alone does not guarantee that routing is "meaningful." A routing policy may achieve high performance by operating over a redundant society where all actors behave identically, or by assigning queries inconsistently to different actors based on superficial variations in input. In such cases, the structural conditions required for actors to develop stable specializations are absent.

The paper identifies two orthogonal structural properties necessary for meaningful routing:

Behavioral Differentiation: The society must contain actors that respond differently to queries. If all actors behave identically, routing is vacuous.
Routing Stability: The routing policy must be robust to surface-form variations. Semantically equivalent queries (e.g., those with typos, synonym substitutions, or paraphrases) must be assigned to the same actor. Without this stability, actors cannot reliably specialize.

Methodology

1. Measuring Society Diversity: Hierarchic Social Entropy (HSE)

To quantify behavioral differentiation, the authors adapt Hierarchic Social Entropy (HSE), originally proposed for robot swarms (Balch, 2000), to language model societies.

Behavioral Vectors: Instead of comparing model weights, the method evaluates actors on an evaluation set $E$ to produce behavioral vectors $b_k \in \mathbb{R}^L$ , where each element represents the actor's score on a specific prompt.
Distance Metric: Inter-actor distance is calculated using $1 - \text{cosine similarity}$ of these behavioral vectors. This captures profile similarity rather than absolute performance magnitude.
Hierarchical Clustering: Actors are clustered at every possible distance threshold $h$ using single linkage.
Entropy Calculation: Simple social entropy (Shannon entropy) is computed for the partition of actors at each threshold $h$ . HSE is the integral of this entropy across all taxonomic levels (from $h=0$ to $h \to \infty$ ).
Properties: HSE is a continuous ratio measure with an absolute zero (identical actors). It is normalized to allow comparison across societies of different sizes. The metric is designed to be sensitive to the degree of separation between groups, not just the number of groups.

2. Measuring Routing Robustness

To quantify routing stability, the authors introduce a perturbation-based metric $\rho(\pi, Q)$ .

Perturbation Taxonomy: Queries are perturbed across five levels of increasing semantic distance:
1. Character-level (typos, spacing).
2. Word-level (synonyms, misspellings).
3. Syntax-level (passive voice, reordering).
4. Paraphrase (full rewrite, same meaning).
5. Rambling (added irrelevant context).
Metric Definition: For a query $q_i$ routed to actor $a_i$ , the robustness $\rho_i$ is the fraction of its $p$ perturbations that are also routed to $a_i$ . The dataset-level robustness is the average of $\rho_i$ over all queries.
Assumption: If $q_i$ and its perturbation are semantically equivalent, the optimal actor assignment should remain the same. Deviations indicate policy instability.

3. Experimental Setup

The methods were validated on two benchmarks:

EmbedLLM: 112 models evaluated on 36,054 questions across 10 benchmarks (e.g., MMLU, GSM8K).
RouterBench: 11 models evaluated on 36,497 queries across 8 datasets.
Synthetic Societies: To test theoretical bounds, the authors constructed synthetic societies of "experts" with binary performance indicators based on:
- RD (Reasoning Depth): 5 tiers of cognitive complexity.
- SA (Subject Area): 15 distinct academic/practical domains.
- RD+SA: A combined pool with partial overlap.
Routing Policies: Random, Prompted (LLM-based selection using role descriptions), and KNN (k-Nearest Neighbors in embedding space).

Key Results

1. Diminishing Returns in Diversity

Real vs. Synthetic: Purpose-designed specialist societies (SA, RD) exhibit significantly higher HSE than pools of real-world models of equivalent size. For example, the SA society (HSE=3.91) is more than twice as diverse as the default EmbedLLM pool (HSE=1.62).
Coreset Heuristic: HSE exhibits strong diminishing returns. In the default EmbedLLM society, adding agents beyond approximately nine yields negligible diversity gains. In synthetic societies, the curve plateaus even faster (4–5 agents). This suggests a curated subset of fewer than ten agents can recover most available diversity in large pools.

2. The Accuracy-Robustness Trade-off

KNN Routers: KNN-based routers achieve high task accuracy on specialist societies (where behavioral boundaries are sharp). However, they suffer a catastrophic collapse in robustness under perturbation. Small surface-level changes shift query embeddings across domain boundaries, causing the router to assign semantically equivalent queries to different actors.
Prompted Routers: Prompted routing (using natural language descriptions) maintains high robustness across all perturbation types and society configurations. While their clean-task accuracy on specialist societies may be slightly lower than KNN, their perturbed accuracy remains stable. They latch onto semantic intent rather than embedding proximity.

3. Impact of Society Composition

Switching to a max-HSE subset (a small, diverse subset selected greedily) improves the robustness of prompted routers.
The effect on KNN routers is dataset-dependent. On RouterBench, the max-HSE subset rescues KNN robustness. On EmbedLLM, KNN robustness decreases or improves only marginally, suggesting that the specific behavioral structure of the EmbedLLM pool (less structured separation even at max HSE) makes nearest-neighbor routing inherently brittle to surface variations.

Significance and Claims

The paper claims to provide the first behavioral diversity metric for LM routing analysis and a corresponding robustness metric. Its primary contributions are:

Reframing Evaluation: It argues that routing meaningfulness requires both a differentiated society and a stable policy, distinct from task accuracy.
Practical Design Heuristic: The finding of strong diminishing returns in HSE suggests that large, heterogeneous model pools are often redundant. Designing societies with fewer than ten carefully curated agents is a practical strategy to maximize diversity without the cost of scaling.
Diagnostic Insight: The divergence between KNN accuracy and robustness highlights that high performance on clean queries does not imply a robust routing system. Prompted routing is shown to be the more stable approach for maintaining specialization conditions under input variation.

The authors remain modest regarding the scope, noting that their metrics measure assignment stability rather than the actual emergence of specialized competence over time, and that their synthetic societies use idealized binary performance profiles that may be sharper than real-world agents.