When Your LLM Reaches End-of-Life: A Framework for Confident Model Migration in Production Systems

1. Problem Statement

Enterprise organizations increasingly rely on proprietary Large Language Models (LLMs) for critical production systems, such as customer support question-answering (QA) services. A significant operational challenge arises when these underlying models reach their End-of-Life (EOL) or when organizations need to migrate for reasons like cost optimization, latency reduction, or regional availability.

The core difficulties in migrating these systems include:

• Evaluation Bottlenecks: Manual evaluation is too slow and expensive for the short migration windows (often ~12 months) and the breadth of impacted products.
• Metric Misalignment: Standard automated metrics (e.g., ROUGE, BLEU, or generic RAGAS scores) often fail to capture business-specific requirements, such as the preference for a "I don't know" (IDK) response over a confident but incorrect answer, or specific stylistic constraints.
• Prompt Coupling: Prompts optimized for one model often degrade in performance on another, requiring re-tuning which adds complexity.
• Data Scarcity: System-specific test sets are often small, making statistical confidence in automated metrics difficult to achieve.

2. Methodology: The Migration Framework

The authors propose a six-stage framework designed to balance evaluation efficiency with rigorous quality assurance. The framework was applied to a commercial QA system serving 5.3 million monthly interactions across six global regions, originally powered by Claude 3 Haiku.

Stage 1: Candidate Filtering

• Vetting: Models must pass internal checks for bias, privacy, compliance, and licensing.
• Schema Compliance: Models are eliminated if they cannot consistently generate the required output schema (e.g., structured XML with specific flags for IDK).

Stage 2: Metric Calibration (The Core Contribution)

To address the unreliability of automated metrics on small, domain-specific datasets, the authors introduce a Bayesian statistical approach:

1. Manual Grounding: A small subset of the test set is manually evaluated by human assessors to establish a "Gold Standard."
2. Confusion Matrix Construction: Automated metrics are compared against human judgments to calculate True Positive Rates (TPR) and False Positive Rates (FPR).
3. Bayesian Inference: Using uninformative priors, the authors estimate posterior distributions for TPR and FPR.
4. Monte Carlo Sampling: These distributions are used to simulate the "true" correctness of models across the full test set, generating 90% confidence intervals for the difference in correctness between the baseline and candidate models. This allows for defensible decisions even with limited data.

Stage 3: Correctness Comparison

Candidates are filtered based on the Bayesian analysis. Models showing a statistically significant decrease in correctness or a significant increase in IDK rates (on answerable questions) are eliminated.

Stage 4: Style and Latency Checks

• Style: Automated substring matching checks for unwanted phrases (e.g., "according to my sources") and measures word count to ensure conciseness.
• Latency: Median response times are compared against the baseline.

Stage 5: Prompt Adaptation

The framework investigates whether adapting the prompt (via manual editing, Amazon Bedrock's Prompt Management, or MIPROv2) can improve performance. The authors found that the baseline prompt generalized surprisingly well, and automated adaptation often increased IDK rates without improving accuracy on diverse test sets.

Stage 6: Final Selection

The final choice is made based on the remaining candidates that satisfy correctness, style, latency, and regional availability constraints, with a final tie-breaker on cost.

3. Key Contributions

1. Bayesian Calibration Framework: A principled method to calibrate automated LLM evaluation metrics against human judgment. This quantifies uncertainty and provides confidence intervals for model comparisons, solving the "small data" problem in enterprise migrations.
2. Domain-Specific Metric Refinement: The development of a custom metric, new_correctness, which evaluates answers against the full source context rather than just the ground truth. This accounts for potential errors or incompleteness in the ground truth labels.
3. Empirical Analysis of Prompt Adaptation: A study demonstrating that for many enterprise QA tasks, the original prompt often transfers well to new models, and automated prompt optimization (like MIPROv2) may not yield statistically significant improvements over the baseline in all contexts.
4. Production-Ready Case Study: A real-world demonstration of migrating a high-volume (5.3M interactions/month) system, validating the framework's scalability and cost-effectiveness.

4. Experimental Results

The framework was tested on a pool of candidates including Gemma 3, Anthropic Claude 4.5, AWS Nova series, OpenAI GPT-OSS, and Qwen3.

• Filtering:
  • OpenAI GPT-OSS models were eliminated for failing to generate valid XML.
  • Nova Micro, Nova Lite, and Gemma 3 were eliminated due to inability to confidently claim improvement in correctness (confidence intervals included negative values).
  • Claude 3.5 Sonnet was eliminated due to high latency (1.73s vs. 1.19s baseline).
  • Qwen3-32B (Reasoning Mode) was eliminated due to poor style performance (7.5% bad style vs. 2.1% baseline).
• Final Selection:
  • AWS Nova 2 Lite and Qwen3-32B (with reasoning disabled) were selected as the primary replacements.
  • Performance: Both models offered equal or better correctness than the baseline, with Nova 2 Lite providing significant latency improvements (0.59s vs. 1.19s) and full regional coverage.
  • Cost: The migration resulted in reduced financial costs while maintaining quality.
• Metric Performance: The new_correctness metric showed the highest alignment with human judgment (TPR ~0.905, FPR ~0.375 on the basic test set), outperforming standard RAGAS metrics.

5. Significance and Impact

This paper provides a critical roadmap for enterprises managing portfolios of AI-powered services. As the LLM ecosystem evolves rapidly, models will frequently reach EOL. The proposed framework offers:

• Risk Mitigation: By using Bayesian confidence intervals, organizations can avoid migrating to models that appear better on average but are statistically indistinguishable or worse in specific critical dimensions.
• Cost Efficiency: It reduces the need for massive manual evaluation teams by leveraging calibrated automated metrics.
• Reproducibility: The methodology is agnostic to the specific model or task, provided a definition of "correctness" exists, making it applicable to RAG systems, chatbots, and content generation tools across various industries.

The study concludes that while automated metrics are imperfect, calibrating them against human judgment and applying Bayesian inference creates a robust, defensible process for model migration that balances speed, cost, and quality.