Technical Summary: SOMA – Efficient Multi-turn LLM Serving via Small Language Model

Problem Statement

Large Language Models (LLMs) are increasingly deployed in multi-turn dialogue settings where maintaining conversational context is essential. Standard serving practices concatenate the full dialogue history at every turn to ensure coherence. While reliable, this approach incurs substantial costs in latency, memory, and API expenditure, particularly when queries are routed to large proprietary models. Existing solutions struggle to balance response quality with efficiency:

• Single-model approaches (e.g., compression, memory retrieval, attention reuse) still rely on the large model for every turn, failing to reduce monetary costs or GPU usage significantly.
• Multi-model approaches (e.g., routing simple queries to smaller models) often fail to generalize across dialogue complexity and introduce overhead from model switching. Furthermore, LLMs rely heavily on early turns for grounding; simply handing later turns to a small model without modeling the accumulated context leads to response drift and degraded quality.

The core challenge is to achieve an efficient, context-aware multi-turn serving framework that avoids recomputing the full history at every turn while preserving response quality.

Methodology: SOMA

The authors propose SOMA (Soft-prompts for lOcal Manifold Approximation), a framework that exploits the "long-tail" distribution of token counts in multi-turn dialogues (where early turns are information-dense and later turns are shorter updates). SOMA adapts a smaller surrogate model ($G$) to approximate the behavior of a large original model ($F$) within a constrained local region of the reasoning manifold defined by the early conversation.

The framework operates via a three-stage pipeline:

1. Soft Prompt Tuning (Mining Weak Alignment Directions)

The system uses the early turns of a session to identify where the surrogate model $G$ diverges most significantly from the large model $F$.

• Objective: Learn a soft prompt matrix $P$ that maximizes semantic divergence between $F$ and $G$.
• Loss Function: The objective combines:
  • Semantic Divergence Loss: An unlikelihood-style loss that discourages $G$ from assigning high probability to tokens produced by $F$, extended to a "semantic neighborhood" to catch paraphrased disagreements.
  • Expectation-Weighted Divergence: A term that upweights penalties if the expected embedding of $G$'s output still aligns with $F$'s token, preventing the model from simply spreading probability mass over semantically similar but incorrect tokens.
  • Anti-Degeneration Regularizer: An entropy term to prevent the soft prompt from collapsing into repetitive or bland outputs.
• Process: The soft prompt is applied to $G$'s embedding layer, while a nearest-neighbor verbalization is applied to $F$ to ensure comparable conditioning.

2. Localized Fine-Tuning (LoRA Adaptation)

Once the soft prompts identify the "hard" local cases (directions of maximal disagreement), these examples are distilled into a lightweight adaptation of the surrogate.

• Data Selection: Turns with the highest mined divergence scores are selected to form a local training set.
• Adaptation: The surrogate $G$ is fine-tuned using LoRA (Low-Rank Adaptation) on attention and MLP projections, optimizing for the original model's responses on these selected turns.
• Result: The continuous soft prompts are discarded. The adapted surrogate is now specialized to the local manifold of the current session.

3. Efficiency Inference with Switching and Rollback

• Switching: A semantic gate (based on cosine similarity) determines when to switch from the large model $F$ to the adapted surrogate $G$. Switching occurs only after a warm-start window and an acceptance batch confirm that the surrogate is aligned with the original model and the dialogue remains within the learned local state.
• Inference: Post-switch, the system serves turns using the adapted surrogate with a compressed input (a fixed-budget summary of early content plus recent turns), avoiding full-history processing.
• Rollback: A drift detector monitors the dialogue. If the conversation shifts topic or moves away from the learned local state, the system rolls back to the large model $F$, refreshes the local summary, and re-evaluates the switch condition.

Key Contributions

1. Empirical Discovery of Long-Tail Patterns: The authors reveal a consistent long-tail pattern in multi-turn dialogues across diverse domains (ShareGPT, medical, negotiation, role-play), where early turns carry heavy context and later turns are shorter updates. This motivates the use of a cheaper surrogate for later turns given the accumulated context.
2. SOMA Framework: A novel method for efficient multi-turn serving that:
   • Learns soft prompts to expose maximal response dissimilarity between surrogate and original models.
   • Adapts the surrogate via localized LoRA fine-tuning on mined weak-alignment directions.
   • Enables prompt-free inference with a simple cosine gate for switching and a drift-aware rollback mechanism.
3. Theoretical Analysis: The paper provides theoretical bounds for:
   • Switching Reliability: Concentration bounds for the warm-start generalization and acceptance batch detection.
   • Prompt Search Coverage: Guarantees on the number of soft-prompt candidates needed to cover the local discrepancy manifold.
   • Suboptimality Limits: Analysis of the directional quality of selected candidates.

Experimental Results

Extensive experiments were conducted on six datasets (ShareGPT, ReMeDi, Craigslist, Multi-Char, MATH, MT-Bench) using LLaMA and Qwen model families.

• Response Similarity: SOMA achieves the highest response similarity to the original large model across all datasets, outperforming baselines like History-Prefix (feeding full history to the small model), History-FT (standard fine-tuning), and RouteLLM.
• Task-Grounded Quality: On the MATH dataset, SOMA significantly narrows the accuracy gap between the small surrogate and the original large model, demonstrating that the similarity gains translate to preserved reasoning capabilities.
• Efficiency:
  • Token Usage: SOMA reduces average token usage by avoiding repeated full-history serving after the switch.
  • Break-Even: While short sessions may not benefit due to warm-start overhead, medium-to-long sessions show significant net gains in latency and token costs.
• Ablation Studies: Removing the anti-degeneration regularizer or the expectation-weighted term leads to performance drops, confirming the necessity of stable prompt mining and disagreement-aware selection.
• Reliability: Drift-aware rollback successfully recovers performance when topic shifts occur, maintaining high similarity while minimizing false rollbacks.

Significance and Claims

The paper claims that session-local surrogate serving is a practical and effective direction for efficient multi-turn LLM deployment. By treating multi-turn dialogues as a local manifold approximation problem, SOMA allows small models to replicate the reasoning processes of large models within specific conversational contexts.

The authors emphasize that this approach:

• Reduces serving costs and latency without sacrificing coherence.
• Is particularly effective for sessions where later turns remain tied to an established dialogue state (the "local" condition).
• Mitigates the risk of quality degradation through semantic acceptance gates and automatic rollback mechanisms.

The work suggests that for many real-world interactive applications, the computational burden of re-processing full history with a large model is unnecessary if the system can dynamically adapt a smaller model to the local conversational context.