CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR

The paper introduces CLIPO, a method that integrates contrastive learning into policy optimization to generalize Reinforcement Learning with Verifiable Rewards (RLVR) by capturing invariant structures across correct reasoning paths, thereby mitigating hallucinations and improving the generalization and robustness of Large Language Models.

The Gist

Here is an explanation of the CLIPO paper, translated into simple language with everyday analogies.

The Big Problem: "The Lucky Guess"

Imagine you are teaching a student to solve a complex math problem.

• The Old Way (RLVR): You give the student a problem. They write down a long, messy solution. At the very end, they write the correct answer. You say, "Great job! You got the right answer!" and give them a gold star.
• The Flaw: The student might have gotten the right answer by pure luck, or by copying the answer from a cheat sheet, even though their reasoning was full of nonsense. Because you only checked the final answer, the student learns that hallucinating (making things up) is fine as long as the result looks right. They don't learn how to think; they just learn how to guess the right ending.

The Solution: CLIPO (The "Group Study" Approach)

The authors of this paper propose CLIPO (Contrastive Learning in Policy Optimization). Instead of just looking at the final answer, CLIPO looks at the entire journey of the thinking process.

Here is how it works, using a few analogies:

1. The "Group Study" Analogy

Imagine you are in a study group with 16 students trying to solve the same hard math problem.

• The Old Way: You only check who got the right answer. If Student A got it right, you praise them. You don't care that Student A's logic was weird.
• The CLIPO Way: You look at all the students who got the right answer. You notice that even though they wrote different things, they all shared a specific, logical "core" in their thinking.
  • The Insight: "Hey, Student A, Student B, and Student C all got it right. If you look closely, they all used this specific logical step. That's the secret sauce."
  • The Action: CLIPO forces the AI to realize that the "secret sauce" (the correct reasoning pattern) is what matters, not just the final number. It tells the AI: "Stay close to the other smart students who got it right, and stay far away from the students who got it wrong."

2. The "Dance Floor" Analogy

Think of the AI's thinking process as a dance floor.

• The Problem: In the old method, anyone who ended up at the "Correct Answer" corner of the room got a prize, even if they stumbled, tripped, and danced wildly to get there.
• The CLIPO Fix: CLIPO installs a new rule. It says, "If you want a prize, you don't just need to reach the corner; you need to dance in a similar rhythm to everyone else who reached that corner."
  • If two people got the right answer but danced completely differently (one used logic, the other guessed), CLIPO pushes them apart.
  • If two people got the right answer and used the same logical steps, CLIPO pulls them closer together.
  • This creates a "safe zone" of good reasoning. The AI learns to find the "invariant structure"—the common, logical path that all successful attempts share.

3. The "Noise Cancellation" Headphones

Imagine the AI's brain is full of static noise (hallucinations, random guesses, bad logic).

• Old Method: The noise doesn't matter as long as the final song sounds right.
• CLIPO: CLIPO acts like high-end noise-cancelling headphones. By comparing many "successful" attempts against each other, it filters out the random noise (the unique mistakes each student made) and amplifies the clear signal (the shared, correct logic).
  • It effectively says: "Ignore the weird steps you took. Focus on the steps that everyone who succeeded took."

Why This Matters (The Results)

The paper tested this on hard math problems (like those in high school competitions).

• Before: The AI would sometimes get the right answer but fail when the question was slightly changed (e.g., changing a number or the wording). It was "overfitting" to the specific answer.
• After CLIPO: The AI became much more robust. Because it learned the underlying logic rather than just memorizing answers, it could handle:
  • Perturbed tasks: Questions where the numbers are changed or scrambled.
  • Symbolic tasks: Abstract problems that require pure logic.
  • Generalization: It didn't just get better at math; it got better at reasoning in general, without needing humans to grade every single step of its thinking.

Summary

CLIPO is like upgrading a teacher from a "Grade-Only" system to a "Peer-Review" system.
Instead of just checking if the answer is right, it looks at the group of students who got it right, finds the common thread in their thinking, and teaches the AI to follow that thread. This stops the AI from "cheating" with lucky guesses and forces it to learn how to think clearly and consistently.

The Bottom Line: It makes AI smarter, more reliable, and less likely to hallucinate, by teaching it that how you get the answer is just as important as what the answer is.

Technical Summary

Here is a detailed technical summary of the paper "CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR".

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become a dominant paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs), particularly in mathematics and coding. Methods like Group Relative Policy Optimization (GRPO) rely on outcome-based rewards (binary signals: correct/incorrect) derived from external verifiers (e.g., code compilers, math checkers).

However, the authors identify a critical limitation in standard RLVR:

• Neglect of Intermediate Steps: RLVR treats all trajectories leading to a correct final answer as equally valuable, regardless of whether the intermediate reasoning steps were logical or hallucinated.
• Overfitting and Hallucination: Models may learn to "guess" the answer or memorize ground-truth outcomes without understanding the underlying logic (answer-copying). This leads to poor generalization, especially on perturbed, symbolic, or out-of-distribution (OOD) tasks.
• Lack of Fine-Grained Supervision: Existing solutions like Process Reward Models (PRMs) require expensive human annotations for step-level feedback, while entropy-based methods often fail to capture semantic logical importance.

The core problem is how to provide dense, informative, and robust supervision during policy optimization without relying on costly human process annotations.

2. Methodology: CLIPO

The authors propose CLIPO (Contrastive Learning in Policy Optimization), a framework that integrates contrastive learning into group-based policy optimization to generalize RLVR.

Core Insight

Successful reasoning paths for a given problem share an invariant logical structure, whereas incorrect paths or hallucinations manifest as sporadic, uncorrelated noise. By maximizing the similarity between successful trajectories in an embedding space, the model can distill this shared logical essence.

Technical Architecture

1. Intra-Group Contrastive Head:

   • A lightweight auxiliary head ($g_\phi$) is appended to the LLM backbone.
   • It takes the last hidden states of the generated trajectories and projects them into a semantic embedding space.
   • Sentence-Level Representation: Token-level hidden states are aggregated via mean pooling to form a single trajectory embedding.

2. Contrastive Objective (InfoNCE):

   • Within a rollout group $G = \{y_1, ..., y_G\}$ for a prompt $x$, trajectories are categorized based on the binary verifier reward $r(x, y)$.
   • Positives: Correct trajectories ($r=1$).
   • Negatives: Incorrect trajectories ($r=0$).
   • The model optimizes an InfoNCE loss to maximize the similarity between pairs of correct trajectories while minimizing similarity with incorrect ones.
   • Mathematically, this maximizes the mutual information among positive rollouts, enforcing them to cluster in the embedding space.

3. Reward Integration:

   • The contrastive loss is converted into a dense auxiliary reward signal ($r^{CL}$).
   • The final reward for a trajectory $y_i$ is:
     $$r'_i = r_i + r^{CL}_i$$
     where $r_i$ is the original sparse binary reward, and $r^{CL}_i$ is derived from the contrastive loss (scaled by a coefficient $\lambda$ and clipped to prevent dominance).
   • This creates a dense reward landscape that guides the policy toward logically consistent paths even among multiple correct answers.

3. Key Contributions

• Generalization of RLVR: CLIPO extends RLVR beyond coarse outcome supervision by introducing a contrastive mechanism that captures the invariant structure of successful reasoning.
• Dense Reward Shaping: It transforms sparse binary feedback into a dense, relational signal that distinguishes between "good" correct paths and "lucky" or hallucinated correct paths.
• No Human Annotation: Unlike Process Reward Models (PRMs), CLIPO requires no additional human-labeled step-level data; it leverages the existing outcome verifiers and the inherent structure of successful rollouts.
• Robustness to Distribution Shift: The method explicitly targets the "overlap" of successful paths, making the model more robust to perturbations and symbolic variations where standard RLVR often fails.

4. Experimental Results

The authors evaluated CLIPO on two tracks using diverse models (Qwen2.5, Llama3.1, DeepSeek-R1-Distill) and benchmarks.

Track I: Grade-School & General Reasoning (GSM8K)

• Setup: Fine-tuned on GSM8K, evaluated on GSM8K variants (Symbolic, P1, P2) and general QA benchmarks (CommonsenseQA, TruthfulQA, MMLU).
• Results:
  • GRPO + CLIPO achieved the highest average score (63.26), outperforming all baselines (GRPO, GSPO, DAPO, GMPO).
  • Significant gains were observed on GSM8K-P2 (+3.36 points) and GSM8K-Symbolic (+0.58 points), demonstrating superior robustness to distribution shifts.
  • Improved performance on general reasoning tasks without sacrificing general knowledge.

Track II: Competition-Level Reasoning (MATH, AMC, AIME)

• Setup: Trained on MATH 7.5k, evaluated on MATH500, Math-Perturb, AMC23, AIME, and AIME25.
• Results:
  • DAPO + CLIPO achieved the best average score (44.05).
  • CLIPO consistently improved performance across all base RLVR methods (GRPO, GSPO, DAPO, GMPO) by +0.80 to +1.35 on average.
  • Notable improvements on Math-Perturb Hard (+0.59) and AIME (+5.42 for DAPO+CLIPO), highlighting exceptional robustness to problem perturbations.

Ablation Studies

• Contrastive Head: Freezing the contrastive head (CLIPO-fixed) led to performance drops, confirming that the head must be jointly optimized to learn the specific semantic manifold for reasoning.
• Loss Variants: InfoNCE performed best, followed by SupCon and SoftNN, indicating the robustness of the contrastive approach across formulations.
• Temperature ($\tau$): Lower temperatures (e.g., 0.02) yielded better results, suggesting that sharper similarity scaling helps the model focus on hard negatives.
• Group Size: Larger group sizes (e.g., 32 rollouts) provided richer contrastive signals and better performance.

5. Significance

CLIPO represents a significant step forward in scalable and robust reasoning for LLMs.

• Paradigm Shift: It moves RLVR from a purely outcome-driven approach to one that implicitly learns the process of reasoning by analyzing the geometric relationships between successful trajectories.
• Scalability: By avoiding the need for human-labeled process rewards, CLIPO offers a scalable solution for improving reasoning in complex domains like mathematics, code generation, and agent planning.
• Generalization: The results demonstrate that enforcing consistency among successful paths effectively suppresses hallucinations and improves generalization to unseen, perturbed, or symbolic tasks, addressing a major bottleneck in current LLM reasoning capabilities.

The code and training recipes are open-sourced, facilitating further research into contrastive policy optimization.