🤖 machine learning

Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls

This paper demonstrates that the widely used sign-branching repetition penalty in LLM inference engines is fundamentally flawed because it depends on the arbitrary logit zero-point, causing massive token selection instability across models and catastrophic failures in structured JSON output, a problem that is resolved by applying the penalty to normalized log-probabilities instead of raw logits.

Original authors: Peter Hollows

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Peter Hollows

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot writer that loves to tell stories. Sometimes, it gets stuck in a loop, repeating the same word over and over like a broken record. To fix this, engineers gave the robot a "repetition penalty" knob. The idea is simple: if the robot tries to say a word it just used, turn up the knob to make that word less likely, forcing the robot to move on.

For years, almost every robot writer in the world (from Hugging Face to vLLM to llama.cpp) has been using the exact same type of knob. But this paper reveals a shocking secret: this specific knob is broken because it's looking at the wrong number.

The Broken Compass

To understand the glitch, imagine the robot's brain is a giant map where every possible word has a score. Positive scores mean "I really want to say this," and negative scores mean "I really don't." The map is flexible; the robot can slide the whole map left or right (adding a constant number to every score) without changing which word it picks next. It's like moving a whole city on a map; the streets are still in the same order relative to each other, even if the city is now in a different country.

The broken knob tries to decide how much to punish a repeated word by checking if its score is positive or negative.

If the score is positive, it divides the number by a penalty factor (making it smaller).
If the score is negative, it multiplies the number by the penalty factor (making it even more negative).

Here's the problem: The line between "positive" and "negative" is arbitrary. Because the robot can slide its entire map left or right without changing its behavior, a word that is "positive" on one robot might be "negative" on another, or even on the same robot if you shift the map slightly.

The paper proves that this "sign-branch" (checking if the number is positive or negative) is looking at a coordinate that the robot's training never actually fixed. It's like trying to drive a car by steering based on whether the dashboard is currently in the "red" or "blue" zone, when the whole dashboard can slide back and forth on its own.

The Chaos of "Same Knob, Different Results"

Because of this sliding map, the same setting on the knob (like 1.3) does completely different things to different robots.

On one robot (like gpt2), a setting of 1.3 might punish 84% of repeated words.
On another robot (like Qwen2.5-Coder-7B), that same 1.3 setting might punish almost none of them, or punish them in a totally different way.

The authors tested this by taking a single robot and sliding its map left and right. They found that with a standard setting of 1.3, the robot changed its mind on 58% to 96% of its choices just because the map slid a tiny bit!

If you use a "subtractive" penalty (just subtracting a number), the robot stays the same.
If you use the standard "multiplicative" sign-branch, the robot goes crazy, changing its mind on nearly every single word it picks.

This means that if you tune your robot to work well with a setting of 1.3, you are actually tuning it to work with a specific, accidental "zero-point" of that specific robot. If you move that robot to a different computer or update its software, that same setting might break it completely.

The JSON Disaster

The most dangerous part of this bug is that it destroys "structured output." This is when you ask the robot to write code or JSON (a specific format for data) that must follow strict rules, like closing every curly brace { or adding a comma ,.

These rules require the robot to repeat certain symbols. But because the penalty looks at the raw numbers, it often mistakes a "necessary rule" for a "bad repetition."

The paper tested this on 200 real-world JSON schemas.
With the standard setting of 1.3, the rate of valid, working JSON output crashed from 97% down to just 23%.
The robot started breaking the grammar, forgetting to close brackets or missing commas, because the penalty was too aggressive on the wrong numbers.

This wasn't a simulation guess; the authors measured this directly on five different models, including code-specialized ones, and saw the same crash happen in every major software stack (Hugging Face, vLLM, and llama.cpp).

The Fix: Normalize First

The paper offers a simple, proven fix. Instead of looking at the raw, sliding numbers, the penalty should look at the normalized probabilities (the actual percentage chance of a word being picked).

Probabilities are always between 0 and 1, so they don't slide around.
When the authors applied the penalty to these normalized numbers instead of the raw scores, the chaos vanished.
The "flip rate" (how often the robot changed its mind just because of a slide) dropped to 0.00.
The JSON success rate at 1.3 bounced back up to 97%.

Hugging Face already has a tool called LogitNormalization that does this, but it's turned off by default and runs after the penalty. The paper suggests that if you turn it on and run it before the penalty, you get a stable, reliable result that works the same way on every model.

The Bottom Line

The "multiplicative repetition penalty" that is currently shipped in almost every AI engine is fundamentally flawed because it relies on a number line that can slide around. It's not a universal knob; it's a broken dial that gives you different results on every machine.

What it does: It causes massive, unpredictable changes in what the AI writes (flipping 58–96% of choices) and breaks structured formats like JSON (dropping validity from 97% to 23%).
What it isn't: It's not a feature; it's a bug that has been copied across the industry for years.
The solution: Normalize the numbers first. This removes the sliding map problem and makes the penalty behave consistently, no matter which robot you are using.

The authors measured this across multiple models and software stacks, and the results are clear: the current standard is broken, and the fix is ready to go.

Technical Summary: Gauge Dependence and Structured-Output Corruption in Sign-Branched Repetition Penalties

Problem Statement
The paper identifies a fundamental flaw in the multiplicative repetition penalty widely deployed across the LLM inference ecosystem (including Hugging Face, vLLM, and llama.cpp). The standard operator, inherited from the CTRL model, applies a penalty to previously seen tokens by branching on the sign of their raw logits: dividing positive logits by a factor $\theta$ and multiplying negative logits by $\theta$ .

The core issue is that the location of the logit zero-point is arbitrary. The softmax function is shift-invariant ( $\text{softmax}(z+c) = \text{softmax}(z)$ ), meaning a model's training objective constrains the probability distribution but leaves the absolute values of the logits (the "gauge") unconstrained. Consequently, the sign-branch of the penalty operates on a coordinate that the training process does not fix. This leads to two measurable negative consequences:

Undefined Behavior: A fixed repetition penalty setting (e.g., $\theta=1.3$ ) performs a different mathematical operation on different models (or even the same model with a shifted bias), as the fraction of tokens falling on the "divide" vs. "multiply" branch varies wildly based on the model's logit zero-point.
Structured-Output Corruption: The penalty indiscriminately suppresses tokens that are required by grammar (e.g., closing braces in JSON, indentation in code) simply because they have appeared before. This causes a catastrophic drop in the validity of structured outputs.

Methodology
The authors conduct a series of controlled experiments across five models (ranging from gpt2 to Qwen2.5-7B, including base and RLHF variants) and two inference stacks (vLLM and llama.cpp) to isolate the effects of the sign-branch.

Gauge Flip Experiment (Consequence 1): The authors decode the same model twice with identical random seeds and inputs, but with a constant $c$ added to all logits (simulating a shift in the model's bias). For a well-defined intervention, the greedy token selection should remain identical. They measure the "flip rate" (fraction of positions where the token differs) at $\theta=1.3$ compared to controls (subtractive penalties and normalized penalties).
Structured Output Evaluation (Consequence 2): The authors evaluate the penalty's impact on two tasks:
- HumanEval: Generating code where specific delimiters are required.
- JSONSchemaBench: Generating JSON objects against 200 real-world schemas.
  They measure the rate of valid, schema-conformant output and the frequency of "delimiter flips" (where a required structural token is incorrectly suppressed).
Cross-Stack Verification: The experiments are replicated in vLLM and llama.cpp using their native samplers to confirm the issue is inherent to the operator's implementation across the ecosystem.
Proposed Fix: The authors test a variant where the penalty is applied to normalized log-probabilities ( $\log p_i$ ) rather than raw logits ( $z_i$ ), effectively removing the sign-branch since all log-probabilities are non-positive.

Key Results

Gauge Dependence: The standard sign-branch penalty is highly sensitive to the logit zero-point.
- Across five real checkpoints, the fraction of seen tokens with positive logits (and thus divided) ranges from 0.17 (gpt2) to 0.95 (Qwen2.5-Coder-7B).
- In the controlled flip experiment, shifting the logit zero-point by a constant of $\pm 5$ causes the standard penalty to flip 58% to 96% of greedy tokens at $\theta=1.3$ . In contrast, subtractive penalties and normalized penalties show a 0% flip rate.
- The "cost" of the intervention (measured in log-likelihood) varies by a factor of 4.1 depending on the gauge, whereas controls remain constant.
Structured Output Failure: The penalty severely degrades structured generation.
- On JSONSchemaBench, applying $\theta=1.3$ drops the rate of valid JSON output from 97% to 23%.
- On HumanEval, the penalty causes flips on structural tokens with a balanced accuracy of 0.999 against a theoretical prediction, confirming the mechanism is driven by the penalty suppressing necessary delimiters.
Ecosystem Prevalence: The sign-branch implementation is found in Hugging Face, vLLM, llama.cpp, and twelve other engines (e.g., TGI, SGLang, Ollama). The issue is not isolated to a single library but is a systemic artifact of the operator's design.
Effectiveness of Normalization: Applying the penalty to normalized log-probabilities ( $\log \text{softmax}(z)$ $lo g softmax (z)$ ) eliminates both effects.
- Gauge flip rate drops to 0.00.
- JSON validity at $\theta=1.3$ recovers to 97%.
- Delimiter corruption decreases by approximately 80x.

Significance and Claims
The paper argues that the default multiplicative repetition penalty is not a "clean scalar knob" but an inference-time intervention whose behavior is partially determined by an unconstrained training artifact (the logit zero-point).

Lack of Portability: A fixed $\theta$ value does not represent the same operation across different models, making tuning per-model and per-task necessary even for a "standard" parameter.
Unintended Consequences: The penalty corrupts structured output by treating grammatical necessities (which must repeat) as unwanted repetitions, a failure mode that is particularly acute for code and JSON generation.
Solution: The authors demonstrate that normalizing logits before applying the penalty (or using subtractive penalties) removes the gauge dependence and the corruption of structured output. They note that Hugging Face already ships a LogitNormalization operator, but it is currently disabled by default and applied after the penalty, which fails to solve the problem.

The paper concludes that while repetition penalties are necessary workarounds for model behavior, the current standard implementation introduces "sharp edges" that compromise reliability. The authors call for stack maintainers to either adopt the subtractive family of penalties or reorder the multiplicative penalty to operate on normalized log-probabilities.