The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility
This paper reveals that the choice of inference backend acts as a critical, yet often unreported, hyperparameter that can significantly alter LLM benchmark scores by up to 16.6 percentage points due to system-level optimizations, thereby urging the community to standardize inference stack reporting to ensure reproducibility.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a judge in a high-stakes cooking competition. You have five identical recipes (the AI models) and five different chefs (the inference backends). You expect that if the recipes are the same, the dishes should taste exactly the same, right?
This paper argues that the chef matters just as much as the recipe.
In the world of Large Language Models (LLMs), researchers usually focus on the "recipe" (the model's weights and training). They assume that if you feed the same prompt into the same model, you get the same answer. This paper reveals that this assumption is broken. The "chef" (the software engine running the model) is a silent, invisible variable that can completely change the outcome of the dish, sometimes making a great recipe taste terrible, or a mediocre one taste amazing.
Here is a breakdown of what the paper found, using simple analogies:
1. The "Silent Hyperparameter"
In AI research, a "hyperparameter" is a setting you tweak to get better results (like temperature or speed). The authors discovered that the inference backend (the software like vLLM, llama.cpp, or Ollama) acts like a hidden hyperparameter that nobody talks about.
- The Analogy: Imagine two people reading the same book. One reads it in a quiet library (the standard software), and the other reads it while riding a bumpy rollercoaster (a high-speed, optimized engine). Even though the book is identical, the rollercoaster rider might skip a word, misread a sentence, or get distracted, leading them to tell a different story at the end.
- The Finding: The authors tested 5 different "chefs" (engines) on 5 different "recipes" (models). They found that simply switching the engine could change a model's test score by up to 16.6 percentage points. That is a massive swing—enough to make a model jump from "average" to "world champion" or vice versa, even though the model itself didn't change.
2. The "Butterfly Effect" in Conversation
AI models generate text one word at a time. If the engine makes a tiny mistake in the very first word, the whole conversation can go off the rails.
- The Analogy: Think of a game of "Telephone." If the first person whispers a slightly different word than intended, the last person hears something completely different.
- The Finding: The paper showed that these engines often disagree on the very first few words of an answer. For complex reasoning tasks (like solving math problems), one engine might start the solution correctly, while another engine starts with a tiny error. That tiny error cascades, causing the engine to generate a completely different, and often wrong, chain of thought.
3. Why Does This Happen? (The Kitchen Secrets)
The authors dug into the code to find out why the chefs were making different dishes. They found two main reasons:
- Reason A: Hidden Defaults (The "Secret Sauce"): Some engines have hidden settings that turn on automatically.
- Example: One engine (Ollama) secretly adds an extra "start" token to the prompt, like adding a pinch of salt you didn't ask for. Another engine (LMDeploy) forces a specific penalty on repeating words. When the researchers turned these "secret sauces" off, the scores jumped back up, proving the engine was messing with the results.
- Reason B: Speed Tricks (The "Fast-Forward" Glitch): To make AI run faster, engineers use mathematical shortcuts (like grouping calculations or using lower precision math).
- Example: Imagine doing a long math problem. One person does it step-by-step with a calculator (standard). Another person uses a super-fast mental math trick that rounds numbers slightly differently. The final answer is usually close, but sometimes that tiny rounding error changes the result. In AI, these "speed tricks" cause tiny floating-point errors that pile up and change the final answer.
4. The "Invisible" Problem in Research
The authors surveyed over 35,000 research papers to see if scientists were admitting which "chef" they used.
- The Finding: Almost no one reported it. It's like a cooking competition where contestants say, "I used a secret recipe," but refuse to say which stove they cooked it on.
- The Consequence: Because researchers don't report the engine, we can't tell if a new "State-of-the-Art" model is actually better, or if it just got lucky because it was tested on a specific engine that happened to boost its score. This makes scientific progress hard to verify.
5. Safety Risks
The paper also tested safety. They asked the models to try to "break out" of their safety rules (jailbreaks).
- The Finding: A model that is safe and refuses to answer a dangerous question on one engine might suddenly become vulnerable and answer it on a different engine. The "deployment gap" means a model tested as safe in a lab might be unsafe in the real world just because the software running it is different.
The Bottom Line
The authors are calling for a new standard in AI research: Stop treating the software engine as invisible.
Just as you would report the temperature and the oven type when baking a cake, researchers must report exactly which inference engine they used. Until we do that, we can't be sure if we are comparing apples to apples, or if we are just comparing apples to apples that were sliced by different knives.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.