Technical Summary: Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving

Problem Statement

Efficient deployment of Large Language Models (LLMs) in production environments necessitates a trade-off between accuracy and serving cost. Operators typically default to a single model: either a large, expensive model that over-invests compute on easy queries, or a smaller, efficient model that underperforms on difficult ones. Existing model routing systems often require annotations beyond standard task evaluation (e.g., human preference data or complex difficulty labels) and struggle to jointly optimize routing under an explicit inference cost budget while recovering accuracy through post-generation quality estimation. Furthermore, many approaches do not adapt easily to changes in the available model pool without manual reconfiguration.

Methodology

The authors propose a two-stage cascaded framework that jointly optimizes routing under a Time Per Output Token (TPOT) budget and recovers accuracy using only task-correctness labels.

Stage 1: Clustering-Based Routing

The first stage operates at the cluster level to pre-route queries, avoiding the inefficiency of running an efficient model on every query before deciding to escalate.

1. Semantic Clustering: Incoming queries are encoded using all-MiniLM-L6-v2 and partitioned into coherent groups ($N$ clusters) using k-means. The number of clusters is selected by maximizing the mean Silhouette score.
2. Cost-Aware Routing: For each cluster $c$ and model $m$, a routing score is calculated:
   $$\text{Score}(m, c) = \text{Error}(m, c) + \lambda \cdot \text{Cost}_{\text{norm}}(m)$$
   Where $\text{Error}(m, c)$ is the empirical error rate on training data, and $\text{Cost}_{\text{norm}}(m)$ is the normalized TPOT of the model (0 for the fastest, 1 for the slowest).
3. Hyperparameter Tuning ($\lambda$): An interpretable hyperparameter $\lambda$ controls the accuracy-latency trade-off. The optimal $\lambda^*$ is automatically tuned offline on training data to satisfy a user-specified TPOT budget $B$.
4. Pareto Analysis: Before routing, models that are Pareto-dominated (slower and less accurate than another model across all clusters) are pruned from the pool.
5. Routing Decision: Each query is assigned to the nearest cluster centroid and routed to the model minimizing the score for that cluster.

Stage 2: Quality Estimation (QE) Cascade

The second stage acts as a safety net for queries routed to efficient models.

1. Mechanism: A lightweight binary classifier (fine-tuned ModernBERT-base) inspects the output of efficient models.
2. Input: The classifier takes the query, the model's output, and the generation length.
3. Decision: Based on task-correctness labels derived from the efficient model's performance on training data, the classifier decides to either accept the output or escalate the query to a stronger model.
4. Integration: This stage is applied only to outputs from efficient models; queries routed directly to strong models in Stage 1 bypass the classifier to preserve latency savings.

Key Contributions

• Two-Stage Framework: A novel architecture combining pre-generation cluster-level routing with post-generation quality estimation, addressing the "multi-stage cascade gap" identified in prior work.
• Cost-Aware Optimization: The system jointly optimizes routing under an explicit TPOT budget using a single, interpretable hyperparameter ($\lambda$), rather than sweeping a monetary cost curve.
• Minimal Annotation Requirement: The framework requires only standard task-correctness labels (available from standard benchmark evaluations) for both clustering-based routing and QE classifier training.
• Dynamic Adaptability: The system adapts to changes in the model pool (addition or removal of models) automatically through Pareto analysis and re-computation of routing tables, without manual reconfiguration.
• Generalization: The approach is validated across two distinct domains: telecommunications (TeleQnA) and mathematical reasoning (AIME 2024).

Experimental Results

The framework was evaluated on TeleQnA and AIME 2024 using pools from the Qwen 3/3.5 and Gemma 4 families.

• AIME 2024 (Mathematical Reasoning):

  • The combined Stage 1+2 system achieved 88.4% accuracy with a TPOT of 9.7 ms.
  • This retains 97% of the strongest model's (Qwen3-30B) accuracy (89.1%) while reducing latency by 18% (2.1 ms faster).
  • Stage 2 recovered a 2.0% accuracy gain over Stage 1 alone with only a 0.2 ms TPOT increase.

• TeleQnA (Telecommunications QA):

  • The combined system achieved 74.3% accuracy with a TPOT of 23.8 ms.
  • This is within 2.1% of the strongest model's accuracy (76.4%) while being 0.7 ms faster.
  • Stage 2 recovered 3.1 percentage points of accuracy over Stage 1 alone.

• Extensibility: Experiments adding new models to the pool demonstrated that the framework automatically prunes dominated models and integrates superior ones, achieving Pareto improvements without manual intervention.

Significance and Claims

The paper claims that this framework offers a practical, interpretable solution for deploying LLMs in cost-constrained environments, particularly for autonomous agents in sectors like telecommunications where on-premise deployment and low latency are critical.

• Efficiency vs. Accuracy: The system demonstrates that operators do not need to choose between a single strong model or a single efficient model. Instead, a cascaded approach can retain near-state-of-the-art accuracy while significantly reducing inference costs (TPOT).
• Operational Simplicity: By relying on task-correctness labels and an interpretable hyperparameter ($\lambda$), the framework provides operators with a "simple path" for managing model pools, avoiding the complexity of training complex routers on preference data.
• Scalability: The use of Pareto analysis ensures that the routing logic remains robust as model pools evolve, making the framework suitable for dynamic production environments where new models may become available.

The authors acknowledge limitations, noting that the framework currently relies on offline clustering (not adapting to query distribution shifts at inference time) and uses TPOT as a proxy for cost, which does not account for queueing or network overhead. However, they posit that the framework provides a foundational, cost-aware architecture for efficient LLM serving.