🤖 AI

Continual Model Routing in Evolving Model Hubs

This paper addresses the challenges of scaling model selection in rapidly expanding AI hubs by formalizing the Continual Model Routing (CMR) setting, introducing the large-scale CMRBench benchmark, and proposing CARvE, a contrastive embedding method that significantly outperforms existing baselines in routing accuracy across diverse models and tasks.

Original authors: Jack Bell, Giacomo Carfì, Gerlando Gramaglia, Vincenzo Lomonaco

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jack Bell, Giacomo Carfì, Gerlando Gramaglia, Vincenzo Lomonaco

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a massive, ever-expanding library. Every day, thousands of new books (AI models) arrive, written by different authors, covering every topic imaginable. You have a specific question, and you need to find the one perfect book to answer it.

In the past, you might have asked a librarian to read a few books to see which one was best. But with millions of books, reading even a few takes too long and costs too much. You need a super-fast librarian who can look at your question and instantly point to the single best book without opening any of them first.

This paper is about building that super-fast librarian for the world of AI models, and it solves a specific problem: What happens when the library keeps growing while you are trying to find books?

Here is the breakdown of the paper's ideas, using simple analogies:

1. The Problem: The Library Never Stops Growing

Currently, AI model hubs (like Hugging Face) are like libraries that never stop adding new shelves.

The Challenge: If you train a librarian to know the books from 2023, they will be confused when 5,000 new books arrive in 2024.
The "Catastrophic Forgetting" Issue: If you try to teach the librarian about the new books, they often forget the old ones. It's like studying for a new math test and suddenly forgetting how to do basic addition.
The Scale: The library isn't just adding a few books; it's adding thousands. A simple "search" isn't fast enough, and a "read everything" approach is too slow.

2. The Solution: "CARvE" (The Smart, Adaptable Librarian)

The authors propose a new system called CARvE (Continual Anchored Router with Contrastive Embeddings). Think of CARvE as a librarian who uses a special "memory anchor" system.

The "Anchor" Metaphor: Imagine the librarian has a set of heavy, permanent anchors tied to the books they already know. When new books arrive, the librarian learns about them without pulling up those anchors. This ensures the old books stay exactly where they were in the librarian's mind, while the new books find their own spot.
The "Map" (Embeddings): Instead of reading the whole book, CARvE creates a tiny "map coordinate" (an embedding) for every book based on what it does. When you ask a question, the system just checks which coordinate is closest to your question. It's like checking a GPS location rather than reading a map.
The "Practice" (Replay): To make sure the librarian doesn't forget the old books, the system occasionally shows them a small, carefully selected sample of old questions and answers. It's like a teacher giving a quick pop quiz on old material to keep the memory fresh.

3. The Test: "CMRBench" (The Simulation)

To prove their librarian works, the authors built a giant simulation called CMRBench.

They created a fake library that grows over time, simulating four different "eras" of model releases.
It includes over 2,000 different models (books) across many topics.
They tested their librarian against other methods (like just searching by title or trying to retrain the whole system from scratch).

4. The Results: Why CARvE Wins

The paper shows that CARvE is much better than the other methods for three main reasons:

It Doesn't Forget: While other methods got confused and forgot the old models as new ones arrived, CARvE kept its memory of the old books intact.
It's Fast and Efficient: It doesn't need to read the whole library to find the answer. It uses the "map coordinates" to find the right book instantly.
It Handles the "Family" Problem: Sometimes, you don't need the exact same book, just a book from the same "family" (e.g., a specific version of a model). CARvE is good at finding the right family of models, not just the exact title, which is often more useful in real life.

5. What They Didn't Do (Important Limits)

The paper is very clear about what this system cannot do yet:

No "Zero-Shot" for New Books: If a brand new book arrives that the librarian has never seen and has no examples of, CARvE cannot instantly know how to route it. It needs to see a few examples of how that new book is used before it can add it to its map.
It's Pre-Selection: This system picks the book before you read it. It doesn't read the book to check if the answer is good; it just guesses which book is most likely to have the right answer based on the title and description.

Summary

In short, this paper says: "We built a smart system that can manage a giant, growing collection of AI models without forgetting the old ones."

They created a benchmark to test this (CMRBench) and a method called CARvE that uses "anchors" to keep old knowledge stable while learning new things. It works better than trying to retrain the whole system from scratch or just searching by keywords, making it a practical solution for the future of AI where models are constantly changing.

Technical Summary: Continual Model Routing in Evolving Model Hubs

1. Problem Definition

The rapid expansion of AI model hubs (e.g., Hugging Face) has shifted the central challenge from building models to selecting the appropriate pre-trained model for a specific query. While Mixture-of-Experts (MoE) architectures and orchestration pipelines have demonstrated the value of routing, existing approaches face two fundamental limitations when applied to real-world, evolving hubs:

Scalability: Routing must scale to thousands of candidate experts, rendering exhaustive post-inference evaluation or naive ranking infeasible.
Non-Stationarity: Model hubs are dynamic; new experts are added, existing ones are updated, and others become obsolete. Static retrieval methods (e.g., ranking by model card similarity) are brittle in large, evolving spaces, and standard retrieval-augmented training (RAT) does not ensure stable routing performance as the target space shifts.

The authors formalize this setting as Continual Model Routing (CMR), treating pre-inference model selection as a continual classification task where the label space (the set of available models) grows over time. The goal is to design a router that can incrementally adapt to new models while retaining competence on previously seen ones, without retraining from scratch.

2. Methodology: CARvE

To address CMR, the authors propose CARvE (Continual Anchored Router with Contrastive Embeddings). Unlike baselines that rely on model cards (natural language descriptions) or execute candidate models, CARvE routes based on embedding similarity between a user query and a learned registry of model embeddings.

Core Components

Embedding-Based Router: The router uses a frozen backbone language model (e.g., LLaMA2-7B) with LoRA adapters trained at each experience. It projects tokenized queries into a fixed-dimensional embedding space ( $\mathbb{R}^d$ ) and compares them against a registry of learned model embeddings using temperature-scaled cosine similarity.
Asymmetric Anchoring: To mitigate catastrophic forgetting, CARvE employs a checkpoint-based anchoring mechanism. At the start of each experience $t$ , a snapshot of the router parameters (projection head and model embeddings) from $t-1$ is taken. During training on $E_t$ , the loss penalizes drift for embeddings of previously seen model IDs, while embeddings for newly introduced models remain unanchored and free to adapt. This preserves the geometric stability of the existing registry.
Candidate-Set Training: To avoid the computational cost of a full softmax over thousands of models, training is performed on fixed-size candidate sets ( $K$ ). These sets include the ground-truth model, hard negatives (mined from the current pool), semantic negatives (same domain), and far negatives (different domains).
Structured Replay: The method utilizes a domain–model coreset replay buffer. Instead of random sampling, it allocates replay quotas per domain and uses Farthest-Point Sampling (FPS) within domains to select diverse examples, ensuring coverage of the long-tail distribution of models.

3. Benchmark: CMRBench

The authors introduce CMRBench, the first benchmark specifically designed for continual model routing.

Structure: It simulates realistic hub expansion through four sequential experiences, integrating APIBench, ToolMMBench, and a new dataset, HuggingBench.
Scale: The benchmark spans over 2,000 candidate models across multiple domains.
Metrics: In addition to standard Model-ID and Domain accuracy, the authors introduce Model Family Accuracy. This intermediate-granularity metric groups functionally similar variants (e.g., different quantization levels of the same architecture) to better capture semantic routing quality without being overly strict on exact model identifiers.

4. Experimental Results

Extensive experiments compare CARvE against strong baselines, including zero-shot fine-tuning, retrieval-only methods (BM25, BGE-M3), model merging (TIES, DARE), and various continual learning strategies (EWC, LwF, random replay).

Performance: CARvE significantly outperforms all baselines. At a 10% replay budget, CARvE achieves 80.7% Domain Accuracy with only 5.9% Forgetting, compared to 75.9% accuracy and 13.1% forgetting for standard random replay. At 20% replay, it reaches 82.9% accuracy with 3.0% forgetting.
Comparison to Upper Bounds: CARvE narrows the gap to joint training (training on all data simultaneously) and, in some domain-level metrics, surpasses it while operating under the stricter continual constraint.
Ablation Studies:
- Anchoring: Disabling projection anchoring causes a significant drop in accuracy and a spike in forgetting, confirming its critical role in stabilizing the embedding space.
- Replay Strategy: Domain-stratified coreset replay outperforms random replay, particularly at lower replay budgets (5-10%).
- Initialization: Random initialization of model embeddings outperforms warm-starting from model card embeddings, as card text encodes linguistic similarity rather than routing discriminability.
Efficiency: CARvE reduces computational costs (TFLOPs and GPU hours) by approximately 45–60% compared to cumulative or from-scratch retraining, making it viable for large-scale deployment.

5. Significance and Claims

The paper claims to be the first work to empirically demonstrate the central role of continual learning in model routing through large-scale benchmarks and analysis. Its primary contributions are:

Formalization: Defining CMR as a class-incremental learning problem where the router must scale to thousands of experts and adapt to non-stationary pools.
Benchmarking: Providing CMRBench as a reproducible reference point for the community, moving beyond static candidate pools.
Methodology: Demonstrating that CARvE, through contrastive embedding, structured replay, and asymmetric anchoring, enables scalable and robust expert selection.

The authors emphasize that their approach addresses a practical systems bottleneck: selecting a specialist model before inference to reduce latency and cost. They note that while CARvE reduces reliance on storing raw user histories, it does not eliminate privacy risks, and better routing could inadvertently amplify biases if not paired with safety policies. The work suggests that the future of model selection lies not in monolithic models, but in ecosystems of evolving, interacting models managed by continual learning systems.