💬 NLP

RouteProfile: Elucidating the Design Space of LLM Profiles for Routing

This paper introduces RouteProfile, a comprehensive design space for LLM profiles structured across four key dimensions, and demonstrates through systematic evaluation that structured, query-level profiles significantly enhance routing performance and generalization across diverse router mechanisms.

Original authors: Jingjun Xu, Hongji Pu, Tao Feng, Haozhen Zhang, Jiaxuan You, Ge Liu

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingjun Xu, Hongji Pu, Tao Feng, Haozhen Zhang, Jiaxuan You, Ge Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a manager at a massive call center. You have a huge team of employees (the LLMs), but they aren't all the same. Some are great at math but terrible at creative writing. Others are experts in history but struggle with coding. Your job is to take a customer's question (the Query) and instantly send it to the employee who can answer it best. This is called LLM Routing.

For a long time, researchers have been obsessed with building better "dispatchers" (the Routers) to make these decisions. But they mostly ignored the Employee Profiles (the LLM Profiles)—the actual descriptions of what each employee is good at.

This paper, RouteProfile, asks a simple but powerful question: Does the way we write these employee profiles matter more than the dispatcher itself?

Here is the breakdown of their findings using simple analogies:

1. The Problem: Bad Profiles Lead to Bad Decisions

Imagine trying to hire a doctor.

The Old Way (Flat Profiles): You just write a name tag: "Dr. Smith." That's it. The dispatcher has no idea if Dr. Smith is a heart surgeon or a podiatrist.
The New Way (Structured Profiles): You write a detailed resume that connects Dr. Smith to their medical school, their specific training, and the types of patients they've successfully treated in the past.

The paper argues that most current systems use the "Name Tag" approach. They treat models as simple, flat lists. The authors built a new system called RouteProfile to test if using a "Resume" approach (structured information) makes the dispatcher smarter.

2. The Four Ways to Write a Profile

The authors created a "menu" for designing these profiles, looking at four ingredients:

Organization: Is the info just a messy pile of notes (Flat), or is it a connected web showing how skills relate to each other (Structured)?
Format: Is the profile written in plain English (Text) or converted into a secret code of numbers that computers understand better (Embeddings)?
Depth: Does the profile only look at the employee's direct skills, or does it also look at their colleagues, their training history, and the projects they worked on (Aggregation Depth)?
Learning: Is the profile static (written once and never changed), or does it learn and improve as it sees more data (Trainable)?

3. The Big Discoveries

The researchers tested these different profile styles against three different types of dispatchers. Here is what they found:

Structure Wins: A messy pile of notes (Flat) is almost always worse than a connected web of information (Structured).
- Analogy: It's like trying to navigate a city using a list of street names versus using a map that shows how the streets connect. The map (Structured Profile) helps the dispatcher find the right path every time.
Details Matter More Than Generalizations: Profiles that focus on specific, small tasks (Query-level) work better than profiles that just say "Good at Math" (Domain-level).
- Analogy: Telling a dispatcher "This employee is good at solving quadratic equations" is much more useful than just saying "This employee is good at Math." The specific detail prevents mistakes.
The "New Employee" Test (Generalization): What happens when you hire a brand new employee you've never seen before?
- Analogy: If you have a new hire, a static profile won't help. But if you have a Structured Profile that was Learned (trained on data), the dispatcher can look at the new hire's connections (e.g., "They went to the same school as our best coder") and guess they are good at coding, even without seeing them work yet.
- Result: Flat profiles failed completely with new models. Structured, trainable profiles were the only ones that could successfully route tasks to new, unseen models.

4. The Conclusion

The paper concludes that we have been focusing too much on building better dispatchers and not enough on writing better resumes.

If you want your AI system to pick the right model for the job, you can't just rely on the router's brain. You have to give the router a structured, detailed, and connected profile of the models it is choosing from. Without a good profile, even the smartest router will get lost.

In short: Don't just build a smarter traffic cop; give them a better map.

1. Problem Statement

As the Large Language Model (LLM) ecosystem expands, individual models exhibit heterogeneous capabilities across different queries, tasks, and domains. This necessitates LLM routing, a mechanism to select the most suitable model for a given query.

However, existing research predominantly focuses on optimizing router mechanisms (e.g., learning-based, graph-based, or contrastive routers) while largely neglecting the design of LLM profiles. LLM profiles are representations of a model's capabilities derived from interaction histories. Current profiling approaches are often:

Heterogeneous and Entangled: Mixed with routing strategies, making it difficult to isolate the source of performance gains.
Limited in Granularity: Ranging from simple index-based one-hot vectors (semantically impoverished) to coarse benchmark summaries (discarding fine-grained signals) or LLM-generated text (often narrow in coverage).
Unstructured: Failing to integrate the complex, interdependent relationships between models, tasks, domains, and specific query instances.

The core research question is: How does the design of LLM profiles affect routing performance across different routers?

2. Methodology: The RouteProfile Framework

The authors propose RouteProfile, a general framework that treats LLM profiling as a structured information integration problem over heterogeneous interaction histories. They formalize the data sources as a Heterogeneous Interaction Graph ( $G$ ) and define a design space characterized by four key dimensions.

A. Data Representation: The Interaction Graph

The graph $G = (V, E)$ integrates four primary data sources:

Model Family: Architectural lineage and developer priors.
Domain Coverage: Specialization areas (e.g., Math, Coding).
Task Evaluation: Standardized benchmark scores (e.g., MMLU, HumanEval).
Query-Level Instances: Specific problem instances associated with tasks.

Nodes: Model, Model Family, Domain, Task, Query.
Edges: Model-Family, Model-Task, Task-Domain, Task-Query.
Features: Textual descriptions (generated by LLMs like GPT-4o) for nodes and performance scores for edges.

B. The Four Design Dimensions

The framework characterizes profile construction via the aggregation function $f(\omega, \gamma, K, \ell)$ :

Organizational Form ( $\omega$ ):
- Flat: Direct concatenation of data into text or vectors without graph structure.
- Structured: Utilizing the graph topology (e.g., via Graph Neural Networks) to model relational information.
Representation Type ( $\gamma$ ):
- Text: Profiles expressed as natural language summaries (often via LLM prompting).
- Embedding: Profiles expressed as dense vectors (via PLMs or GNNs).
Aggregation Depth ( $K$ ):
- The number of hops in the graph used for information propagation (0 to 4). Determines whether the profile relies on direct neighbors or higher-order context.
Learning Configuration ( $\ell$ ):
- Training-free: Static aggregation (e.g., prompting an LLM or fixed GCN).
- Trainable: Optimized via self-supervised learning (e.g., masked reconstruction) on the interaction graph.

C. Experimental Setup

Datasets: 15 datasets for graph construction (Knowledge, Reasoning, Math, Coding) and 12 for downstream evaluation.
Models: 25 LLMs total (8 candidates for routing, 17 auxiliary for graph context).
Routers Evaluated:
1. SimRouter: Similarity-based, non-parametric (baseline).
2. MLPRouter: Learning-based (MLP projection).
3. GraphRouter: Graph-structured GNN router.
Settings: Standard routing and Cold-Start (generalization to a new, unseen LLM).

3. Key Contributions

Systematic Design Space: Introduced RouteProfile, the first comprehensive framework to decouple profile design from router design, defining four orthogonal dimensions for profiling.
Structured Information Integration: Formalized LLM profiling as a heterogeneous graph problem, moving beyond flat feature extraction to relational integration.
Empirical Disentanglement: Conducted a large-scale systematic evaluation across three distinct router architectures to isolate the impact of profile design choices.

4. Key Results & Findings

RQ1: Impact of Profile Design on Routing Quality

Structured > Flat: Structured profiles (utilizing graph topology) consistently outperform flat baselines across all routers and representation types. This indicates that retaining structural relationships is crucial for informative profiles.
Depth Matters (Context-Dependent): Increasing aggregation depth (hops) generally helps in training-free settings. However, in trainable settings, excessive depth can lead to over-smoothing, degrading performance in MLPRouter and GraphRouter, though it benefits SimRouter.

RQ2: Effectiveness of Data Sources

Query-Level > Domain-Level: Signals at the query level (fine-grained interaction instances) are significantly more reliable and informative than coarse domain-level summaries.
Noise in Domains: Adding domain nodes often fails to improve performance and can introduce noise, weakening otherwise strong profiles.

RQ3: Generalization to New LLMs (Cold-Start)

Structured + Trainable is Critical: Generalization to newly introduced models (Cold-Start) is nearly impossible with flat profiles (near-zero success).
Synergy: The best cold-start performance is achieved by combining structured profiles with trainable configurations. This combination allows the router to leverage relational structure and learned integration to infer capabilities for unseen models.
Co-Design: Generalization gains are not uniform; they depend on the co-design of the profile and the router (e.g., GraphRouter benefits most from structured profiles).

5. Significance and Conclusion

This work shifts the paradigm of LLM routing research from a sole focus on router mechanism design to a balanced view that prioritizes LLM profile design.

Theoretical Insight: It demonstrates that the quality of the input representation (the profile) is a bottleneck for routing performance, often more so than the router architecture itself.
Practical Guidance:
- Avoid flat, index-based, or coarse domain-only profiles.
- Prioritize structured, query-level data integration.
- For robust generalization to new models, employ trainable, structured profile construction.
Future Direction: The paper establishes LLM profiling as a critical, underexplored research direction, offering a principled foundation for developing more effective and generalizable multi-LLM routing systems.