🤖 machine learning

A Geometric Perspective on Stabilizing Value Conflict Resolution

This paper proposes that incorporating value conflict-focused Chain-of-Thought reasoning stabilizes Large Language Model training by geometrically smoothing the loss landscape, thereby resolving optimization instabilities caused by scalar rewards and enhancing moral reasoning performance.

Original authors: Saket Reddy, Andy Liu

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Saket Reddy, Andy Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to be a good friend. You want it to be helpful, honest, and safe all at the same time. But sometimes, being helpful means telling a white lie, while being honest means telling the hard truth. This is a "value conflict." For a long time, scientists have tried to teach robots by giving them a simple scorecard: "Good job, +1 point!" or "Bad job, -1 point!" This is like trying to teach a complex dance by only saying "faster" or "slower." It often leaves the robot confused, stumbling, and crashing into the furniture because it doesn't know how to balance the steps.

To fix this, researchers are looking at something called "Chain-of-Thought" (CoT). Think of this as asking the robot to whisper its thoughts out loud before it acts. Instead of just jumping to an answer, the robot pauses and says, "Hmm, the user wants X, but that might hurt Y. Let me think about how to handle both." This paper explores what happens inside the robot's brain when we force it to do this kind of thinking, especially when it's stuck between two conflicting values. They use a special mathematical tool to look at the "shape" of the robot's learning process, treating it like a physical landscape with hills and valleys.

The Rocky Cliff vs. The Smooth Valley

Imagine the robot's learning process as a hiker trying to find the lowest point in a vast, foggy mountain range. The goal is to find the "bottom of the valley," which represents the perfect, stable way to answer questions.

When the robot is trained using the old-school method of simple scorecards (called RLHF), the landscape it walks on is a disaster. It's like a jagged, sheer cliff face. The hiker is standing on a tiny, sharp peak. If they take even the tiniest step in the wrong direction, they don't just slide down a little; they plummet off a cliff. In the paper's language, this is a "sharp minimum" with a high "curvature." The robot is unstable, prone to wild swings in behavior, and easily confused when faced with tricky moral dilemmas.

The researchers found that when they added Chain-of-Thought—making the robot think step-by-step—the landscape changed. It was no longer a terrifying cliff. Instead, it became a smooth, wide ravine. The hiker could still find the bottom, but now the ground was gentle. Small steps didn't cause a fall; the robot was much more stable.

But they didn't stop there. They wondered: What if we could design the thinking process even better?

The "Annealing" Analogy: Cooling Down the Chaos

To solve the problem of conflicting values, the authors borrowed an idea from physics called annealing. Imagine a blacksmith forging a sword. If you heat the metal up and then cool it down too fast, it becomes brittle and might shatter. But if you heat it up to let the atoms move around freely (exploring all possibilities) and then slowly cool it down, the atoms settle into a perfect, strong, stable structure.

The team created a new type of thinking process called Annealing CoT. They taught the robot to mimic this physical process in three distinct phases:

High Temperature (The Roam): First, the robot is told to "heat up." It broadly explores the problem, weighing all the conflicting values without rushing to a conclusion. It's like looking at the whole map before picking a path.
Cooling (The Filter): Next, it starts to "cool down." It begins to weigh the trade-offs, applying rules to narrow down the options. It decides which values are more important in this specific situation.
Low Temperature (The Decision): Finally, it reaches "low temperature." The robot has settled into a stable, decisive action based on the narrowed-down options.

What They Found

The results were fascinating. When they measured the "sharpness" of the robot's learning landscape using a mathematical tool called the Hessian eigenvalue (which basically measures how steep the cliffs are), they saw a clear pattern:

The Cliff (Base RLHF): The robot trained only on simple scores had a "top eigenvalue" of about 4083. This is a very high number, meaning the landscape was incredibly sharp and unstable.
The Ravine (Standard CoT): When the robot used standard step-by-step thinking, the number dropped to 1345. The cliff was gone, replaced by a smoother slope.
The Flat Valley (Annealing CoT): When the robot used the new "Annealing" method, the number dropped even further to 1218. This indicated the flattest, most stable valley of all.

In plain English, the more structured the thinking process, the more stable the robot became.

Does It Actually Work?

A stable landscape is great, but does it make the robot smarter? The researchers tested their robots on three different "moral exams" to see how well they handled real-world ethical dilemmas.

The Standard Test: The robot trained with the new Annealing CoT method scored the highest, beating the other models by a noticeable margin. For example, on one test called SafetyBench, it scored 64.00, while the standard model scored 53.67 and the unstable cliff-model scored only 43.67.
The "Maybe" Zone: The standard Chain-of-Thought model did better than the unstable cliff-model, but the improvement was small and sometimes within the "margin of error." This suggests that while just making the robot think is helpful, how it thinks matters a lot.
Scaling Up: The researchers also tested this on a much larger robot (a 9-billion parameter model). Even though they couldn't measure the "landscape shape" for the big one, the results were the same: the Annealing CoT model performed the best, suggesting this trick works even for giant brains.

The Takeaway

This paper suggests that the secret to teaching robots to handle complex moral conflicts isn't just about giving them better scores. It's about designing how they think. By structuring their reasoning to mimic the physical process of cooling down from chaos to order, we can smooth out the jagged cliffs in their learning process. This makes them more stable and better at navigating the tricky balance between being helpful and being honest.

The authors are careful to say this is a "proof-of-concept." They haven't solved every problem in the world, and they note that real life is often more complicated than just two conflicting values. But they have shown a promising new path: if we want robots to be good at moral reasoning, we should stop treating them like simple score-keepers and start teaching them to think like careful, cooling-down blacksmiths.

Technical Summary: A Geometric Perspective on Stabilizing Value Conflict Resolution

Problem Statement

Large Language Models (LLMs) aligned via Reinforcement Learning from Human Feedback (RLHF) often struggle to navigate value conflicts inherent in real-world user requests (e.g., balancing helpfulness with honesty). The dominant paradigm maps human preferences to compressed scalar rewards, which forces models to optimize contradictory constraints simultaneously. This optimization process frequently leads to unstable model behavior, manifesting as flawed moral reasoning and unpredictable safety guardrail failures. While Chain-of-Thought (CoT) reasoning has been shown to improve alignment robustness, the underlying mechanism by which it resolves competing values remains poorly understood.

Methodology

The authors propose a geometric analysis of the loss landscape to understand how CoT stabilizes value conflict resolution. They draw an analogy between unresolved value conflicts in LLMs and geometric frustration in condensed matter physics, where conflicting constraints create a rugged, spiky energy landscape. They hypothesize that CoT acts as a form of "annealing," smoothing this landscape.

Experimental Setup

Data Generation: Using the ConflictScope pipeline, the authors generated prompts containing conflicting value pairs (e.g., fairness vs. objectivity). They created two types of reasoning traces for Supervised Fine-Tuning (SFT):
- Standard CoT: Unstructured step-by-step reasoning.
- Annealing CoT: A structured, three-phase reasoning process mimicking thermodynamic annealing:
  - Phase 1 (High Temp): Broad exploration of conflicting constraints without premature solution collapse.
  - Phase 2 (Cooling): Evaluating trade-offs and applying hierarchical principles to narrow actions.
  - Phase 3 (Low Temp): Converging on a decisive, stable action.
Training: The authors fine-tuned Qwen 3.5 (0.8B) and Llama 3.2 (1B) models using four distinct pipelines:
- Base RLHF: Direct scalar reward optimization without SFT.
- Base SFT: SFT on responses only (no CoT), followed by RLHF.
- Standard CoT: SFT on standard CoT traces, followed by RLHF.
- Annealing CoT: SFT on annealing-inspired CoT traces, followed by RLHF.
- Note: A 9B Qwen 3.5 model was also evaluated for performance scaling, though geometric analysis was omitted due to computational constraints.
Geometric Evaluation: To quantify stability, the authors analyzed the Hessian matrix of the loss function. They computed the top eigenvalue ( $\lambda_{max}$ ), which represents the maximum curvature in the sharpest direction of the loss landscape. A lower $\lambda_{max}$ indicates a flatter, more stable minimum. They visualized these landscapes by perturbing weights along the top two eigenvectors.
Performance Evaluation: Models were tested on three benchmarks:
- MMLU Moral Scenarios: Simple moral classification.
- DecodingTrust Machine Ethics: Complex real-world ethical dilemmas.
- SafetyBench: Safety boundaries on sensitive topics.

Key Results

Geometric Stability (Loss Landscape)

Base RLHF: Exhibited a high top eigenvalue ( $\lambda_{max} \approx 4083$ for Qwen 0.8B), corresponding to a "cliff-like" and unstable loss landscape.
Base SFT: Significantly reduced instability ( $\lambda_{max} \approx 1402$ ), smoothing the landscape into a "ravine-like" structure.
Standard CoT: Further reduced the top eigenvalue ( $\lambda_{max} \approx 1345$ ), indicating a flatter minimum than SFT alone.
Annealing CoT: Achieved the lowest top eigenvalue ( $\lambda_{max} \approx 1218$ ), corresponding to the flattest, most stable minimum in the sharpest direction.
Generalization: These geometric trends (RLHF > SFT > Standard CoT > Annealing CoT in terms of stability) were consistent across the Llama 3.2 (1B) model family.

Performance Benchmarks

Base RLHF: Consistently achieved the lowest performance across all benchmarks.
Base SFT & Standard CoT: Showed improvements over Base RLHF, though these gains were often within the margin of error (±1 SE), suggesting potential but not definitive superiority in all cases.
Annealing CoT: Demonstrated the most significant performance gains, outperforming both Base RLHF and Base SFT by a larger margin across all three benchmarks (e.g., 64.00% on SafetyBench vs. 48.67% for Base SFT in the 0.8B model).
Scaling: The performance benefits of the Annealing CoT design generalized to the 9B Qwen 3.5 model, where it consistently outperformed baselines.

Key Contributions

Geometric Diagnosis: The paper demonstrates that resolving value conflicts directly via scalar RLHF creates an unstable, high-curvature loss landscape, suggesting scalar rewards are ill-equipped for this task.
Mechanism of CoT: It provides geometric evidence that CoT reasoning further smooths the loss landscape in its sharpest direction beyond the effects of standard SFT, mitigating optimization instability.
Structured Reasoning Design: The authors introduce and validate an "Annealing CoT" framework. They show that explicitly designing reasoning dynamics (moving from high-temperature exploration to low-temperature convergence) correlates with flatter minima and significantly better moral reasoning performance.
Generalizability: The findings hold across different model families (Qwen and Llama) and scales (0.8B to 9B parameters).

Significance and Claims

The paper claims that explicitly modifying and improving the design of internal reasoning dynamics offers a promising avenue for advancing pluralistic alignment in LLMs. While reward signal engineering (RLHF) is dominant, this work suggests that the structure of how a model reasons through conflicts is equally critical.

The authors modestly note that while the performance increase for standard CoT was sometimes within the margin of error, the consistent trend and the significant gains from the designed Annealing CoT indicate that value conflict-focused CoT has the potential to be an effective mechanism for better moral reasoning. The work serves as a proof-of-concept that moving beyond scalar rewards to intentionally designed cognitive processes can resolve the "geometric frustration" of conflicting values, leading to more stable and ethically robust models.