💬 NLP

Scalable Prompt Routing via Fine-Grained Latent Task Discovery

This paper proposes a scalable two-stage prompt routing architecture that leverages graph-based clustering for fine-grained latent task discovery and a mixture-of-experts model for task-aware quality estimation, achieving superior performance and cost efficiency across diverse frontier models compared to existing baselines.

Original authors: Yunyi Zhang, Soji Adeshina, Patrick Guan, Ashwin Ganesh, Zhen Han, Vassilis N. Ioannidis, Huzefa Rangwala, George Karypis

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yunyi Zhang, Soji Adeshina, Patrick Guan, Ashwin Ganesh, Zhen Han, Vassilis N. Ioannidis, Huzefa Rangwala, George Karypis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a massive, high-tech restaurant with a menu of 11 different celebrity chefs (the Large Language Models). Some are masters of math, some are poets, some are coding wizards, and others are great at general conversation.

Your goal is to serve every customer the perfect dish (the best answer) without wasting money on the most expensive chef for every single order.

The Problem: The "One-Size-Fits-All" Mistake

In the past, restaurant managers (routing systems) tried to guess which chef to use in two ways:

The Manual Menu: They tried to categorize every order into broad buckets like "Math" or "Writing." But this is too vague. A "Math" order could be simple arithmetic or complex quantum physics. A single "Math" category doesn't tell you which specific chef is best for that specific problem.
The General Manager: They hired one super-manager to look at every order and guess the best chef. But with 11 chefs who are all incredibly talented and very similar in skill, this manager gets confused. They can't spot the tiny differences that make Chef A better for this specific puzzle than Chef B.

The result? You either pick the wrong chef (bad food) or you just pick the most expensive chef for everything (wasting money).

The Solution: FineRouter (The Smart Kitchen)

The authors of this paper built a new system called FineRouter. Think of it as a two-stage, super-smart kitchen management system that doesn't rely on a fixed menu or a single guesser.

Stage 1: The "Secret Menu" Discovery (Finding the Hidden Patterns)

Instead of using a pre-written menu, the system looks at thousands of past orders and discovers its own categories.

How it works: It groups similar orders together based on two things: what the order says (the words) and which chefs historically did the best job on similar orders.
The Magic: It finds "micro-categories" that humans might miss. For example, it might realize that "Telephone area code questions" is a totally different skill set than "Solving algebra equations," even though both are technically "Math" or "Facts."
The Result: It creates a "Secret Menu" of 300+ tiny, specific task types. For each type, it knows exactly which 2 or 3 chefs are the best fit.

Stage 2: The "Specialist Team" (The Mixture of Experts)

Now that the system knows the "Secret Menu," it sets up a specialized team.

The Setup: Imagine a central hub. When a new customer order comes in:
1. The Triage Nurse (Stage 1): Quickly looks at the order and says, "Ah, this is a 'Telephone Area Code' request! That goes to the 'Geography/Telecom' team."
2. The Specialist Chefs (Stage 2): Instead of asking all 11 chefs to cook, the system only wakes up the specific chefs who are experts in that tiny category. It asks them, "How good would you be at this specific task?"
The Hybrid Approach: If the system isn't sure which category the order belongs to, it falls back to a general "all-chefs" estimate. But usually, it uses the specialist team.

The Final Decision: The "Best of Both Worlds" Vote

The system combines two pieces of information to make the final call:

The Group Average: "Historically, for this type of task, Chef X is usually the best."
The Specific Nuance: "But looking at this specific wording, Chef Y might have a slight edge."

It blends these two signals to pick the absolute winner.

Why This Matters (The Results)

The paper tested this system with 11 of the world's most advanced AI models across 10 different types of challenges (from coding to trivia to math).

Better Food: The system picked the right chef more often than any single chef could do alone. It actually outperformed the single "best" model in the pool.
Lower Bill: Because it didn't blindly pick the most expensive chef for every job, it achieved these results for less than half the cost.
Scalability: As more chefs join the kitchen (more AI models), this system gets smarter at finding the tiny differences between them, whereas old systems would just get confused.

The Analogy in a Nutshell

Old Way: Asking a single person to guess which of 11 Olympic athletes is best at a specific sport, or just picking the most famous one.
FineRouter Way: First, analyzing thousands of past games to realize there are 300+ specific "sport niches." Then, creating a team of scouts who know exactly which athlete excels in each niche, and only calling on those specific athletes when a new game starts.

This approach ensures you get the gold medal performance without paying for the entire Olympic team every time you need a single answer.

1. Problem Statement

As Large Language Model (LLM) pools expand to include dozens of frontier models (e.g., Claude, Llama, DeepSeek, Qwen), the performance gaps between them on specific tasks are becoming increasingly narrow. Existing prompt routing methods face two critical limitations in this context:

Coarse-Grained Taxonomies: Manually defined task categories (e.g., "Math," "Coding") are too broad to capture subtle capability distinctions. For instance, a model might excel at symbolic algebra but fail at contextual word problems, yet both are treated as "Math."
Monolithic Routers: Single-model estimators struggle to differentiate fine-grained capability differences across diverse tasks when forced to learn a universal mapping for all prompt types simultaneously.

The core challenge is how to dynamically select the optimal model from a large candidate pool to maximize performance while minimizing cost, specifically when models have nuanced, task-specific strengths rather than global dominance.

2. Methodology: FineRouter

The authors propose FineRouter, a two-stage routing architecture that combines automated fine-grained task discovery with task-aware quality estimation.

Stage 1: Automated Fine-Grained Task Discovery

Instead of relying on manual taxonomies, FineRouter automatically discovers latent task structures from training data.

Graph Construction:
- Semantic Signal: Uses an LLM to generate concise task descriptions for each prompt, creating embeddings.
- Performance Signal: Generates ranked lists of preferred LLMs for each prompt based on a quality function $Q^*$ .
- Edge Weighting: Constructs a sparse graph where edges connect prompts based on the geometric mean of their semantic similarity (cosine similarity of descriptions) and performance similarity (Rank Biased Overlap of model preference lists).
Clustering: Applies iterative Leiden community detection to group prompts into clusters. Each cluster represents a discovered latent task type (e.g., "Formal symbolic mathematics" vs. "Contextual word problems").
Candidate Selection: For each discovered task cluster, the system adaptively selects a small set of top-performing candidate models (reducing the search space) using rank fusion.
Classifier Training: A text classifier is trained to map new incoming prompts to these discovered task types (or an "Other" category).

Stage 2: Task-Aware Dynamic Router (Mixture-of-Experts)

This stage performs the actual quality estimation using a Mixture-of-Experts (MoE) architecture.

Architecture:
- Shared Encoders: A shared prompt encoder and LLM embedding layer.
- General Adapters: $|M|$ MLPs that predict quality scores for all models based on global knowledge.
- Task-Specific Adapters: For each discovered task type $t_i$ , specialized adapters are trained only on prompts belonging to that task. These learn refined patterns for specific model-task affinities.
Inference Strategy:
1. The Stage 1 classifier assigns a prompt to a task type $t_i$ .
2. If assigned, the router activates the task-specific adapters for the pre-selected candidate models of that task, while using general adapters for non-candidates.
3. If unassigned, only general adapters are used.
Aggregation: The final routing score combines the task-level prior (median quality of a model on the assigned task cluster) and the prompt-level estimate (MoE prediction) using a weighted sum:
$Q_{final} = \alpha \cdot \text{norm}(\tilde{Q}_t) + (1-\alpha) \cdot \text{norm}(\bar{Q}_t)$

3. Key Contributions

Automated Task Discovery: A scalable method using graph-based clustering to identify fine-grained latent task types directly from data, eliminating the need for manual taxonomy design.
Task-Aware MoE Architecture: A routing framework that leverages discovered task structures to activate specialized prediction heads, allowing the model to focus on specific task-model affinities.
Superior Cost-Performance Trade-off: Demonstrated ability to outperform the strongest individual model while incurring less than half the inference cost.

4. Experimental Results

Setup: Evaluated on 10 diverse benchmarks (QA, Math, Code, Reasoning) using 11 frontier models (including Llama-4-Maverick, Claude-Sonnet-4.5, DeepSeek-R1, Qwen3-235B).
Performance:
- FineRouter achieved the highest overall quality score (0.652), outperforming all baselines (kNN, MLP, RouteLLM, GraphRouter, IPR) and the strongest single model (Claude-Sonnet-4.5 at 0.621).
- It consistently beat baselines across the majority of specific tasks.
Cost Efficiency:
- FineRouter surpassed the performance of the strongest candidate model (Claude-Sonnet-4.5) while costing less than 50% of its API price.
- The routing distribution showed balanced utilization: 28% to Claude-Sonnet-4.5, 27% to DeepSeek-R1, 23% to Llama-4-Maverick, and 13% to Qwen3-235B, proving the router effectively leverages diverse strengths rather than over-relying on one model.
Ablation Studies:
- Fine vs. Coarse: Using automatically discovered fine-grained tasks (Fine CLS) significantly outperformed manually defined coarse categories (Coarse CLS).
- Two-Stage Necessity: The full two-stage architecture outperformed using Stage 1 (task filtering) or Stage 2 (MoE) alone, confirming that both task-level stability and prompt-level adaptability are necessary.
Case Studies: The system successfully identified niche distinctions, such as separating "symbolic math" (favoring Llama models) from "numerical reasoning in reading comprehension" (favoring DeepSeek), and even discovered unexpected domains like "telephone area code queries."

5. Significance

This work addresses a critical bottleneck in the era of model proliferation: scalability. As model pools grow, manual curation of routing rules becomes impossible. FineRouter demonstrates that:

Latent structures exist: Even among frontier models, there are distinct, discoverable task affinities that are invisible to coarse categorization.
Specialization works: Decomposing the routing problem into task discovery and specialized quality estimation yields better results than monolithic approaches.
Economic Viability: It provides a practical pathway for enterprises to utilize expensive frontier models efficiently, achieving state-of-the-art results at a fraction of the cost by intelligently routing queries to the most cost-effective high-performing model for that specific context.

Limitations: The approach relies on the quality of the LLM used for task description generation and the reward model for supervision. It currently handles text-only inputs and requires training data with responses from all candidate models.