Temperature Fragility and the Conditional Benefits of Truncation Sampling
This paper demonstrates that truncation sampling techniques like top-p and min-p primarily improve large language model accuracy only at high temperatures where performance degrades significantly, offering no benefit over standard temperature sampling at the lower default temperatures typically used in deployed systems.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Large language models are the engines behind the text-generating tools that have become part of daily life. When these models write a sentence, they do not simply recall a fixed answer; instead, they predict the next word, or token, by weighing a vast list of possibilities. At each step, the model assigns a probability to every word in its vocabulary, and a computer program draws one from that list to continue the text. A setting called temperature controls how wild that draw can be. A low temperature makes the model stick to the most obvious, likely words, keeping the output safe and predictable. A high temperature spreads the draw out, allowing the model to pick from the long tail of unlikely words. This can make the text more creative or varied, but it also risks pulling the model toward words it is not confident about, where its guesses are least reliable.
For years, developers have used a safety net called truncation sampling to manage this risk. These methods, such as top-p or min-p, act like a filter that discards the least probable words before the model makes its choice. The idea is that by cutting off the bottom of the list, the model can run at a higher temperature without losing its way. Previous studies suggested these filters offered significant accuracy gains, but they tested these ideas at high temperatures that most real-world systems never use. This left a gap in our understanding: do these filters actually help the models we use every day, or are they only useful when the temperature is pushed to extremes?
A researcher at the University of Edinburgh set out to fill this gap by testing thirteen different open-source models on two standard reasoning tasks. They ran the models through a single, controlled pipeline to ensure that every result came from the decoding method itself, not from differences in the software or the questions. They tested the models at temperatures of 0.7, 1.0, and 1.3, which cover the range where most deployed systems operate. The researcher found that the answer depends entirely on the specific model being used. They discovered that some models are fragile, meaning their performance collapses when the temperature rises even slightly above the default, while others are robust and hold their ground.
The study revealed that six of the thirteen models tested suffered a dramatic drop in accuracy when the temperature moved from 0.7 to 1.3. On one of the difficult tests, these fragile models lost between 17 and 38 percentage points of accuracy. The researcher traced this loss to a specific type of failure: the models did not just give wrong answers; they stopped giving answers altogether. Instead of finishing a thought, the text would run on until it hit a hard limit on length, or it would dissolve into nonsense that the grading software could not read. The other seven models tested did not suffer this fate; they maintained their accuracy across the same temperature range, losing at most 10 points, and often none at all.
This distinction changed everything about the value of the truncation filters. For the robust models, the filters provided no benefit. At the standard temperatures used in practice, applying a filter did not improve accuracy over simple temperature sampling. The researcher measured this carefully and found that any potential gain was so small it could be considered negligible. However, for the fragile models, the filters were a lifeline. When the temperature rose to 1.3, every truncation sampler tested successfully recovered the lost accuracy, bringing the models back to near their original performance levels. The filters worked by preventing the models from wandering into the tail of unlikely words that caused the collapse.
The researcher also found that the point at which a model collapses is not fixed; it can be shifted by how the model was trained after its initial creation. One model that was a different version of a fragile base model lost only half as much accuracy as its parent, suggesting that the training process plays a major role in stability. Furthermore, the study showed that these fragile models eventually collapse even at higher temperatures, but the filters can delay this failure, keeping the model coherent at temperatures up to 2.0.
The findings suggest that the reported benefits of truncation sampling are real, but they are conditional. These tools do not universally improve models; they primarily serve to rescue models that are already struggling with higher temperatures. For the majority of models tested, which remained stable at standard settings, the filters offered no advantage. This implies that for practitioners working on tasks with clear, checkable answers, the choice of model is more critical than the choice of sampler. If a model is robust at the intended temperature, adding a filter changes nothing. If a model is fragile, the filter can recover the loss, but the root cause is the model's sensitivity to temperature, not a flaw in the sampling method itself. The study concludes that at the temperatures systems actually use, model selection determines accuracy, and truncation samplers are a remedy for a problem that only some models suffer.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.