💻 computer science

CALM: A Self-Adaptive Orchestration Approach for QoS-Aware Routing in Small Language Model based Systems

The paper introduces CALM, a self-adaptive orchestration framework based on the MAPE-K loop that dynamically routes user queries to a coordinated fleet of specialized Small Language Models (SLMs) while leveraging caching and scheduling to reduce latency by 40% and energy consumption by 50% compared to single-LLM baselines.

Original authors: Hemang Jain, Divyansh Pandey, Karthik Vaidhyanathan

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hemang Jain, Divyansh Pandey, Karthik Vaidhyanathan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a busy restaurant. In the past, you might have hired one giant, super-expensive chef who could cook anything from sushi to steak to dessert. This chef is incredibly talented, but they are slow, they use a massive amount of gas (energy), and they sometimes get confused if you ask for something very specific, like "a gluten-free, low-sodium dish for a diabetic."

The authors of this paper, CALM, propose a different way to run your kitchen. Instead of one giant chef, you hire a team of six specialized chefs.

One is a master of Italian pasta.
One is a sushi expert.
One is a dessert specialist.
One is a vegan expert.
And so on.

These "Small Language Models" (SLMs) are smaller, faster, cheaper to run, and much better at their specific jobs than the giant chef. But here is the problem: You can't keep all six chefs standing at the stove at the same time because your kitchen is too small (your computer's memory is limited).

CALM is the smart manager that decides which chef should cook which dish, right in the moment, to make sure you get your food fast, cheap, and delicious.

Here is how the CALM manager works, broken down into simple steps:

1. The Smart Menu (Model Registration)

Before any orders come in, every chef writes down a short description of what they are good at.

Chef A says: "I am great at medical advice."
Chef B says: "I am great at legal contracts."
Chef C says: "I am great at fitness tips."

The manager (CALM) keeps a list of these descriptions.

2. The Order Taker (Routing)

When a customer walks in and says, "I have a sore throat and a fever," the manager doesn't just pick a random chef.

The Brain Scan: The manager quickly reads the customer's request and compares it to the chefs' descriptions. It realizes, "Hey, the Medical Chef is the best fit!"
The Speed Check: But wait, the manager also checks the "live stats." Is the Medical Chef currently busy? Are they moving slowly today? Did they just use a lot of energy on the last order?
The Decision: If the Medical Chef is slow or tired, the manager might say, "Okay, let's send this to the General Health Chef instead, who is currently faster."

This is called Self-Adaptive Routing. The manager constantly learns and changes its mind based on how the chefs are performing right now, not just what they are supposed to do.

3. The Kitchen Shelf (Caching)

Since the kitchen is small, the manager can only keep three chefs at the stove (in the computer's memory) at any one time. The other three are sleeping in the back room (on the hard drive).

If the customer orders "Sushi," and the Sushi Chef is already at the stove, great! The manager sends the order immediately.
If the Sushi Chef is in the back room, the manager has to wake them up and bring them to the stove. This takes a few seconds (a "cold start").
The Magic Trick: The manager is smart about who stays at the stove. If the Sushi Chef just cooked for three people in a row, the manager keeps them there. If the Italian Chef hasn't been called in an hour, the manager sends them back to the back room to make space for the next likely order.

This is the Caching Module. It saves time by keeping the most popular chefs ready, so you don't have to wait for them to wake up.

4. The Feedback Loop (MAPE-K)

The manager doesn't just guess; it watches everything.

Monitor: It watches how long each order takes and how much gas (energy) is used.
Analyze: It asks, "Is the Sushi Chef getting slower?"
Plan: It decides, "Okay, for the next 10 orders, let's prioritize speed over energy," or "Let's prioritize accuracy."
Execute: It changes the rules for the next customer.

What Did They Find?

The authors tested this system against the "Giant Chef" (a single Large Language Model) and found some impressive results:

Faster: The system was about 40% faster (lower latency).
Greener: It used about 50% less energy.
Just as Good: The quality of the answers (confidence) was just as high as the giant chef, sometimes even better for specific topics.
Smarter Memory: By using the "Kitchen Shelf" (caching), they could run the whole system on a much smaller computer (using less memory) without slowing things down too much.

The Bottom Line

The paper argues that instead of trying to make one giant, expensive AI that does everything, we should use a team of smaller, specialized AIs managed by a smart, self-adjusting system. This approach saves money, saves energy, and gets you answers faster, all while keeping the quality high. It's like swapping a single, overworked super-chef for a well-managed team of specialists.

Technical Summary: CALM – A Self-Adaptive Orchestration Approach for QoS-Aware Routing in Small Language Model based Systems

1. Problem Statement

Modern AI-enabled systems face significant runtime uncertainties stemming from dynamic workloads, resource fluctuations, and the inherent non-determinism of autoregressive language models. While Large Language Models (LLMs) offer broad capabilities, they are often resource-intensive, expensive to run via APIs, and prone to hallucinations or "surface-level competence" in domain-specific tasks (e.g., healthcare, law). Conversely, Small Language Models (SLMs) offer efficiency and domain specialization but lack the versatility to handle diverse, shifting contexts when deployed individually.

The core challenge lies in orchestrating a fleet of domain-specialized SLMs to replace a single general-purpose LLM. This requires solving three specific problems:

Domain Accuracy: Ensuring high factual accuracy in critical domains where general LLMs often fail.
Resource Constraints: Managing memory (VRAM) and compute limits, as keeping multiple large models loaded simultaneously is impractical.
Adaptive Routing: Developing a mechanism that dynamically routes queries to the most appropriate SLM based on real-time performance metrics (latency, energy, confidence) and semantic relevance, rather than static load balancing. Existing adaptive approaches often overlook the specific overheads and uncertainties of autoregressive text generation.

2. Methodology: The CALM Approach

The authors propose CALM (Context-Aware Language Model orchestration), a self-adaptive system built on the MAPE-K (Monitor, Analyze, Plan, Execute, Knowledge) feedback loop. CALM integrates two primary components: SORA (Semantic-aware Orchestrator) and CAS (Cache-enabled Adaptive Scheduler).

2.1 System Architecture

Model Registration: SLMs register with SORA by submitting a metadata tuple containing a semantic task description (natural language prompt) and operational attributes (IP, port).
Self-Adaptive Query Routing (SORA):
- Static Scoring: Computes semantic similarity (e.g., Cosine Similarity) between the user query and the model's task description.
- Dynamic Scoring: Incorporates rolling runtime metrics (latency, energy, VRAM usage, confidence) collected by the Monitor.
- Routing Policy: A weighted score combines static and dynamic metrics. The system uses tunable control parameters ( $\lambda$ ) to prioritize specific QoS goals (e.g., Pro-Latency, Pro-Energy, or Pro-Confidence).
- Correlation Handling: The framework acknowledges metric correlation (e.g., latency and energy) and adjusts weights to prevent bias, though the current implementation assumes independent contributions.
Query Processing & Scheduling (CAS):
- Requests are queued in a global buffer.
- The scheduler employs policies such as FIFO (First-In-First-Out) or STCF (Shortest-Time-to-Completion-First) to manage execution order.
Caching Mechanism:
- To address VRAM constraints, CAS maintains a fixed-size cache of models in memory.
- It employs an LRU (Least Recently Used) eviction policy to manage model loading and unloading.
- A "Pro-Cache" routing strategy can bias selection toward models already in the cache to reduce cold-start overhead.

2.2 Experimental Design

The authors evaluated CALM against single-LLM baselines (e.g., Llama-2 13B, Deepseek-MOE 16B) and various ablations of their approach.

Workloads: Simulated 500 user requests across diverse domains (Medical, Fitness, Mental Health, Law, Finance) with varying token limits (256/512) and load rates (12.5–25 RPM).
Metrics: Latency, Energy Consumption (kJ), Confidence Score (1–5), and Peak VRAM usage.
Configurations: Tested variants including Base (static routing), Base + Caching, and Base + Self-Adaptive Routing (with different $\lambda$ priorities).

3. Key Results

The evaluation demonstrates that CALM outperforms single-LLM baselines across multiple dimensions:

Latency & Efficiency: CALM variants reduced latency by approximately 40–50% and energy consumption by 50–60% compared to baseline LLMs. The use of lightweight SLMs was the primary driver for these gains.
Confidence: Domain-specialized routing improved confidence scores by 15–30%, indicating better task alignment and factual accuracy in specific domains.
Memory Optimization: The caching module reduced Peak VRAM usage from ~32 GB (required for full LLM baselines) to as low as 8–19 GB (depending on cache size), enabling deployment on memory-constrained hardware without significant degradation in QoS.
Adaptive Control: The self-adaptive loop allowed for fine-grained trade-offs. For instance, increasing the latency weight ( $\lambda_{lat}$ ) reduced latency by up to 44% while maintaining acceptable confidence. Similarly, "Pro-Cache" routing increased cache hit rates to 83.8%, significantly reducing cold-start penalties.
Scalability: The system maintained stable performance under increasing workloads and generalized effectively across disjoint domains (e.g., Law vs. Medical), confirming the model-agnostic nature of the routing logic.

4. Significance and Claims

The paper claims that CALM offers a practical solution for deploying AI systems under real-world constraints by shifting the paradigm from "bigger is better" to "orchestration is key."

Orchestration over Size: The authors argue that coordinating a fleet of specialized SLMs via adaptive control can outperform single, general-purpose LLMs in terms of efficiency, cost, and domain-specific accuracy.
Handling Uncertainty: By integrating runtime feedback (MAPE-K), the system autonomously adapts to shifting contexts and workload patterns, addressing the inherent uncertainties of language model inference.
Resource Efficiency: The approach enables the deployment of multiple models on constrained hardware (e.g., single GPUs or edge devices) through intelligent caching and scheduling, reducing the environmental and economic footprint of AI inference.
Modularity: The framework supports a modular design where routing, scheduling, and caching can be tuned independently to meet specific Service Level Objectives (SLOs).

The authors conclude that while CALM demonstrates effective single-GPU deployment, future work should explore multi-GPU distributed strategies, resilience to input variations, and expansion to multi-turn conversational settings. They emphasize that their approach provides a foundation for sustainable, cost-efficient, and high-performance AI systems without relying on opaque, resource-heavy API-based LLMs.