💬 NLP

Reward-free Alignment for Conflicting Objectives

This paper proposes **RACO**, a reward-free alignment framework that resolves gradient conflicts in multi-objective LLM training using a novel clipped conflict-averse gradient descent method, achieving superior Pareto trade-offs in tasks like summarization and safety without the need for explicit reward models.

Original authors: Peter L. Chen, Xiaopeng Li, Xi Chen, Tianyi Lin

Published 2026-02-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Peter L. Chen, Xiaopeng Li, Xi Chen, Tianyi Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to create the "perfect" dish. You have two main goals: make it delicious (Helpfulness) and make it healthy (Harmlessness/Safety).

The problem is that these two goals often fight each other. If you want it to be incredibly delicious, you might add lots of butter and sugar (which makes it unhealthy). If you want it to be perfectly healthy, you might serve a plain bowl of steamed broccoli (which isn't very delicious).

In the world of AI, this is called "Conflicting Objectives." When developers try to train an AI to be both helpful and safe, the "instructions" (gradients) often pull the AI in opposite directions, causing it to become unstable or fail at one of the goals entirely.

The Problem: The "Tug-of-War"

Current methods usually try to solve this by just "averaging" the goals. Imagine two people pulling a rope in opposite directions. If you just average their strength, the rope barely moves, and you don't get anywhere. This is what happens to most AI models today—they end up being mediocre at both tasks because they are stuck in a constant tug-of-war.

The Solution: RACO (The "Smart Mediator")

The researchers in this paper proposed a new framework called RACO (Reward-free Alignment for Conflicted Objectives).

Instead of just averaging the two conflicting forces, RACO acts like a Smart Mediator or a Master Chef. Here is how it works using three metaphors:

1. The "Path of Least Resistance" (Conflict-Averse Gradient)

Instead of letting the two forces pull the AI back and forth, RACO looks for a "third way." It searches for a direction where both goals can move forward slightly without one destroying the other. It’s like finding a path through a forest that is both scenic (delicious) and easy to walk (healthy), rather than trying to walk straight through a mountain or straight through a swamp.

2. The "Safety Guardrail" (The Clipping Trick)

The researchers noticed a side effect: sometimes the "Mediator" gets too excited. If one goal is much stronger than the other, the mediator might accidentally pivot the AI too far toward the "loudest" goal, ignoring the user's actual preferences.

To fix this, they added "Clipping." Think of this like a speed limiter on a car or a guardrail on a mountain road. Even if the mediator wants to make a massive, sudden turn to satisfy one goal, the "Clipping" mechanism says, "Whoa, slow down! We have to respect the original balance the user asked for." This keeps the training stable and prevents the AI from "over-correcting."

3. The "No-Recipe" Approach (Reward-Free)

Most AI training requires a "Reward Model"—essentially a separate, expensive AI "judge" that grades the first AI. This is like needing a professional food critic to stand in the kitchen every second to taste the soup.

RACO is "Reward-free." It learns directly from human preferences (the "tastes" of the users) without needing that extra, complicated middleman. This makes it faster, simpler, and more direct.

The Results: A Better Balance

The researchers tested RACO on several famous AI models (like Llama and Qwen) using two main tests:

Summarization: Making summaries that are both high-quality and short.
Safety: Making an AI that is helpful but refuses to help with dangerous or inappropriate requests.

The verdict? RACO consistently won. It produced AI that was better at balancing the "tug-of-war." It didn't just pick a side; it found the "sweet spot" where the AI was both highly useful and reliably safe.

Summary in a Nutshell

Before RACO: AI training was a chaotic tug-of-war where the AI often ended up confused or lopsided.
With RACO: The AI finds a clever middle path, guided by a smart mediator that uses "guardrails" to ensure it stays on track and respects what the human actually wants.

Technical Summary: Reward-free Alignment for Conflicting Objectives (RACO)

1. Problem Statement

Large Language Model (LLM) alignment typically aims to satisfy multiple, often competing, human preferences (e.g., being helpful vs. being harmless, or being informative vs. being concise). Existing alignment methods face two primary challenges when dealing with these conflicting objectives:

Limitations of Linear Aggregation: Most current methods (like DPO or weighted loss functions) use a weighted sum of losses. When objectives have conflicting gradients (i.e., an update that improves one degrades the other), linear aggregation fails to find a direction that satisfies both, often leading to "alignment tax" where one objective is sacrificed for another.
Complexity of Reward Modeling: Many multi-objective approaches rely on explicit reward models for each objective. This adds significant computational overhead, introduces modeling errors, and complicates the training pipeline.

2. Methodology: The RACO Framework

The authors propose RACO (Reward-free Alignment for Conflicted Objectives), a framework that treats alignment as a multi-objective optimization problem using only pairwise preference data (no explicit reward models required).

Key Technical Components:

Conflict-Averse Gradient Descent (CAGrad) Adaptation: Instead of a simple weighted sum of gradients, RACO leverages the principles of CAGrad. It seeks an update direction that maximizes the worst-case local improvement across all objectives, effectively "resolving" the conflict between opposing gradients.
CAGrad-Clip (The Core Innovation): The authors identify that applying vanilla CAGrad to high-dimensional LLM parameter spaces can be unstable. Specifically, the "correction" step in CAGrad can over-correct toward a low-priority objective, violating the user's intended trade-off. To fix this, they introduce clipping: they solve for the optimal correction weights but clip them so they do not exceed the user-specified importance weights ( $w_i$ ).
Mathematical Foundation:
- Convergence: The authors provide theoretical guarantees that the method converges to Pareto-critical points (points where no objective can be improved without degrading another) that respect the user's specified weights.
- Acceleration: They prove that in two-objective cases, the clipping mechanism can strictly improve the convergence rate compared to unclipped versions.

3. Key Contributions

A New Framework (RACO): A reward-free, multi-objective alignment method that directly utilizes pairwise preference data.
Algorithmic Innovation (CAGrad-Clip): A stabilized version of conflict-averse gradient descent that prevents "over-correction" in high-dimensional spaces.
Theoretical Guarantees: Formal proofs of convergence to Pareto-critical points and a proof of training acceleration in two-objective scenarios.
Empirical Validation: Extensive testing across multiple model families (Qwen 3, Llama 3, Gemma 3) and diverse tasks.

4. Experimental Results

The framework was evaluated on two primary tasks: Multi-objective Summarization (Quality vs. Conciseness/Faithfulness) and Safety Alignment (Helpfulness vs. Harmlessness).

Superior Pareto Trade-offs: In all tasks, RACO consistently achieved a better Pareto frontier than baselines like DPO-LW (Weighted Loss) and AMoPO. This means for any given level of "helpfulness," RACO provided higher "harmlessness" than its competitors.
Stability and Control: Unlike baseline methods that often "collapse" one objective to satisfy another (e.g., becoming extremely safe but uselessly unhelpful), RACO maintained a balanced improvement across both metrics.
Robustness across Models: The performance gains were consistent across different model architectures and sizes, including both instruction-tuned and base models.

5. Significance

This paper is significant because it moves LLM alignment away from the "single scalar reward" paradigm toward a more realistic multi-objective paradigm. By providing a mathematically principled and computationally efficient way to handle conflicting goals without the need for complex reward models, RACO offers a scalable solution for developers who need to fine-tune models to meet specific, nuanced, and competing human values. It provides a tool for "steering" models along a desired trade-off curve with higher precision and stability than previously possible.