💬 NLP

Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation

This paper introduces a decoding-time debiasing framework that utilizes a separate Process Reward Model to score and select candidate tokens for fairness and fluency without modifying model weights, demonstrating that sequential critique-and-revise schemes and lightweight bias guards effectively reduce social biases across multiple open-weight and proprietary language models while preserving generation quality.

Original authors: Muneeb Ur Raheem Khan

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muneeb Ur Raheem Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented but slightly biased storyteller (a Large Language Model). This storyteller has read millions of books and articles, and unfortunately, they've picked up some old-fashioned stereotypes along the way. For example, if you ask them to finish a sentence about a "surgeon," they might automatically say "he," or if you ask about a "nurse," they might say "she."

The paper proposes a new way to fix this without rewriting the storyteller's entire brain (which is expensive and hard) or changing the story after it's told. Instead, they introduce a real-time editor who watches every single word as it's being written and says, "Wait, that word might be unfair. Let's try a different one."

Here is how the paper breaks it down, using simple analogies:

The Problem: The "Lazy" Storyteller

The storyteller (the AI) learns from data that contains human biases. Standard fixes involve retraining the AI from scratch or fine-tuning it, which is like sending the storyteller back to school for years. It's expensive, requires special access to the AI's "brain," and might make them worse at other things.

The Solution: The "Decoding-Time" Editor

The authors suggest a different approach: Debiasing at the moment of creation. They don't touch the AI's brain. Instead, they add a separate "Judge" (called a Process Reward Model) that acts like a strict editor.

Every time the AI picks a word, the Judge checks two things:

Fairness: Is this word biased?
Fluency: Does this word sound natural?

If the word is biased, the system intervenes. The paper tests three different ways this editor can work:

1. The "Lottery" Method (Best-of-N)

How it works: The AI quickly thinks of 8 possible words for the next spot. The Judge looks at all 8, picks the fairest one, and uses that.
The Catch: For very smart AI models, this works great. But for smaller, less powerful models, the AI often suggests the same 8 words (or very similar ones) because its "imagination" is limited. It's like asking a small child to pick 8 different colors from a box that only has three crayons; you can't get much variety.
Cost: Surprisingly cheap! If the system is built into the AI's code, the AI only has to "think" once to get all 8 options.

2. The "Critique and Revise" Method (Sequential)

How it works: The AI picks a word. The Judge says, "No, that's biased because it implies surgeons are always men." The AI then thinks, "Oh, I see," and tries again with a better word. They keep doing this until the word passes the test.
The Result: This was the winner in the paper. It works best because it gives the AI a specific reason why a word is bad, rather than just hoping it guesses a good one by luck. It's like a teacher correcting a student's essay with a specific note, rather than just handing back a red "F."
Cost: It takes a bit more time because the AI has to rewrite the word a few times, but it's worth it for the quality.

3. The "Self-Check" Method (Constitutional)

How it works: Instead of an external Judge, the AI is given a list of rules (a "Constitution") and has to check its own work. It asks, "Did I break any rules?" and fixes it if it did.
The Result: This is a good middle ground. It doesn't need an external editor, which is great for privacy or offline use. However, it relies on the AI being smart enough to catch its own mistakes. Smaller models often miss their own errors.

The "Traffic Light" Trick (Bias Guard Gate)

The authors realized that most words in a sentence (like "the," "and," "walked") are totally neutral. Running a full editor check on every single word is a waste of energy.

So, they added a Traffic Light Gate:

The AI suggests a word.
A quick, simple check asks: "Could this specific word be biased in this context?"
Green Light (No): The word goes through immediately.
Red Light (Yes): The full editor (Sequential or Lottery) kicks in to fix it.

This saves a huge amount of computing power. For the smartest AI tested, this gate only turned "Red" 13% of the time, meaning the system stayed fast and efficient.

What They Found

The "Critique and Revise" method was the most effective. It made the AI significantly fairer without making the sentences sound robotic or broken.
Smaller AI models struggled with the "open-ended" stories (writing long paragraphs). They got confused when forced to stop and fix every single word, leading to choppy sentences. The method works best on smarter, more capable models.
Language Matters: They tested this in English and Urdu. While it worked in both, the AI was generally less fluent in Urdu, showing that the method depends on how good the base AI is at the language.

The Bottom Line

You don't need to retrain the AI or have access to its secret code to make it fairer. You can just add a "real-time editor" that watches the words as they are written. The "Critique and Revise" approach is the most powerful, acting like a helpful teacher that guides the AI toward fairness step-by-step, ensuring the stories it tells are both natural and respectful.

Technical Summary: Decoding-Time Debiasing via Process Reward Models

1. Problem Statement

Large Language Models (LLMs) inherit and amplify social biases (regarding gender, race, religion, disability, age, and socioeconomic status) from their training data. These biases manifest in downstream applications, reinforcing harmful stereotypes. Current mitigation strategies face significant limitations:

Training-time methods (e.g., RLHF, DPO, counterfactual data augmentation) require access to model weights, massive curated datasets, and substantial computational resources, making them inaccessible for many practitioners and proprietary models.
Existing inference-time methods often require access to token-level logits (e.g., FUDGE, GeDi), which is impossible with black-box APIs, or rely on prompt engineering, which is unstable and varies with model capability.

There is a need for a debiasing approach that operates at decoding time, requires no weight modification, works with black-box APIs (no logit access), and minimizes computational overhead.

2. Methodology

The paper proposes a framework where a separate Process Reward Model (PRM) acts as a judge to score candidate tokens during generation based on two axes: fairness (bias) and utility (fluency). The framework introduces three distinct debiasing schemes, evaluated in two settings: a controlled single-word fill-in task and an open-ended generation task.

The Three Schemes

Best-of-N (Selection):
- Mechanism: The generator produces $n$ candidate tokens (the top- $n$ from the softmax distribution). The PRM scores each, and the highest-scoring token is selected.
- Overhead Insight: In a native implementation (where the decoder has access to the top- $n$ logits), this requires zero extra generator forward passes ( $R_G = 1$ ), as the candidates are already computed. The cost is only in the $n$ judge calls.
Sequential Critique-and-Revise:
- Mechanism: The generator proposes a token. If the PRM flags it as biased (score below threshold $\tau$ ), the PRM generates a specific critique (e.g., "this word reinforces a gender stereotype"). The generator then revises the token based on this feedback. This loop repeats up to $K$ times or until the token passes.
- Mechanism Advantage: Unlike random sampling, this provides the generator with a concrete signal on why a token is biased and how to fix it.
Constitutional Self-Audit:
- Mechanism: Inspired by Constitutional AI, the generator acts as its own auditor. It checks its proposed token against a hand-written "fairness constitution" (8 principles). If a violation is found, the generator self-critiques and revises.
- Advantage: This scheme is judge-free during the generation loop, making it suitable for environments where external API calls are infeasible.

Extension to Open-Ended Generation: The Bias Guard Gate

To address the high computational cost of applying these schemes to every token in long-form generation, the authors introduce a Bias Guard (BG) gate.

Function: A lightweight binary classifier (single forward pass) asks: "Could this candidate word plausibly carry bias in this context?"
Logic: If the answer is "NO," the word passes immediately. If "YES," the full debiasing scheme (Sequential, etc.) is triggered.
Goal: Exploit the sparsity of bias in natural language (most words are neutral) to keep average overhead near $2\times$ baseline for well-calibrated models.

Overhead Metric

The paper defines a formal overhead metric separating Generator Cost ( $R_G$ ) from Judge Cost ( $R_J$ ). This distinction is crucial for deployment scenarios where the generator runs on-device (edge) and the judge runs in the cloud.

3. Experimental Setup

Models: Four models spanning the capability spectrum:
- GPT-4o-mini: Proprietary anchor (strongest).
- Llama 3.2 (3B), Gemma 3 (4B), Qwen 2.5 (3B): Open-weight models representing compact, on-device LLMs.
Datasets:
- Controlled: 200 prompts (100 English, 100 Urdu) across 8 bias categories.
- Open-Ended: 10 English prompts (20 words each) targeting specific bias categories.
Judge: GPT-4o-mini served as the PRM judge for all experiments.
Languages: English and Urdu (to test transferability to low-resource, morphologically rich languages).

4. Key Results

Single-Word Fill-in

Sequential Performance: Consistently achieved the highest bias scores across all models and languages, improving bias scores by +0.36 to +0.40 over baseline in English and +0.25 to +0.30 in Urdu. Crucially, it preserved or improved fluency (utility scores).
Constitutional Performance: Ranked second, offering a strong judge-free alternative with bias gains of +0.10 to +0.24.
Best-of-N Performance: Surprisingly underperformed on smaller models (Llama, Gemma, Qwen), sometimes performing worse than the baseline. The authors attribute this to candidate collapse, where weaker models produce near-identical top- $n$ tokens, leaving the judge with no diversity to select from. GPT-4o-mini, with a richer distribution, benefited from this scheme.
Urdu Challenges: Utility scores were generally lower in Urdu, reflecting the models' weaker training on this language. However, bias scores were sometimes higher (better) in Urdu, possibly due to models being more conservative when less confident.

Open-Ended Generation

Capability Threshold: GPT-4o-mini showed clear improvements in bias reduction (from 0.46 to ~0.65) while maintaining fluency. Smaller models struggled significantly; forced word-by-word debiasing often broke sentence coherence, causing utility scores to plummet (e.g., Llama 3.2 dropped to 0.14 utility with Best-of-N).
Bias Guard Effectiveness: The gate's firing rate was highly model-dependent. GPT-4o-mini and Qwen fired on only ~13% and ~6.5% of words, respectively, keeping overhead low. Gemma 3 fired on 95.5% of words, negating the optimization entirely. This highlights the need for per-model gate calibration.

Overhead Analysis

Best-of-N: Confirmed to have $R_G = 1$ (native implementation), making it effectively free on the generator side.
Sequential: Proven to be the most cost-effective scheme for bias reduction, achieving the highest bias gains with the lowest generator overhead ( $R_G \approx 1.08$ ) due to its early-stopping mechanism.

5. Significance and Claims

The paper claims the following contributions and significance:

Accessibility: The framework lowers the barrier to bias mitigation. It requires no weight access and no training, allowing any practitioner with API access to apply these schemes immediately.
Novel Application of PRMs: It is the first application of Process Reward Models (previously used for math reasoning) to social bias mitigation, treating each token as a step in a process requiring fairness verification.
Bilingual Benchmark: It provides a controlled bilingual benchmark (English + Urdu) covering 8 bias categories, addressing the English-centric bias in existing literature.
Overhead Accounting: The formal separation of generator and judge costs reveals that Best-of-N is practically free on the generator side in native implementations, a finding often missed in API-based evaluations.
Scalability vs. Capability: The results demonstrate that the debiasing framework scales with model capability. While it works exceptionally well on strong models (GPT-4o-mini), the limitations observed in 3B models are attributed to the base model's fluency and instruction-following capabilities, not a failure of the debiasing framework itself.

6. Limitations Acknowledged by the Authors

Token-Level Patch: The method addresses surface-level stereotypic associations but cannot fix deeper structural biases or world knowledge internalized by the model.
Performative Fairness: There is a risk that models produce text that looks fair by metric standards while underlying associations remain.
Judge Bias: Using a single LLM (GPT-4o-mini) as the judge introduces potential bias and a lack of independence when the same model is the generator.
Lack of Human Evaluation: Scores are model-assigned, not human-validated.
Gate Calibration: The zero-shot Bias Guard gate requires per-model tuning to avoid false positives (as seen with Gemma 3).
Dataset Size: The evaluation sets (200 fill-in, 10 open-ended) are sufficient for trends but not for statistically robust conclusions.

The paper concludes that decoding-time debiasing is a practical, deployable complement to training-time alignment, offering a way to mitigate bias in existing models without gradient updates.