💬 NLP

CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning

CopT is a training-free reasoning framework that reverses the standard chain-of-thought paradigm by first generating a draft answer and then employing continuous-space contrastive verifiers to dynamically trigger on-policy reflection only when necessary, thereby significantly improving accuracy and reducing token costs across diverse reasoning tasks.

Original authors: Dachuan Shi, Hanlin Zhu, Xiangchi Yuan, Wanjia Zhao, Kejing Xia, Wen Xiao, Wenke Lee

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dachuan Shi, Hanlin Zhu, Xiangchi Yuan, Wanjia Zhao, Kejing Xia, Wen Xiao, Wenke Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a brilliant detective trying to solve a mystery.

The Old Way (Chain-of-Thought):
Traditionally, when Large Language Models (LLMs) try to solve a problem, they act like a detective who insists on writing a 10-page police report before they are allowed to say who the culprit is. They think, think, think, write down every step, and then give the answer.

The Problem: Sometimes, the detective already knows the answer in the first second. But because the rules say "think first," they keep writing the report anyway. This wastes time (tokens) and energy, even if the answer was obvious. This is called "performative reasoning"—thinking just for the sake of thinking.

The New Way (CopT):
The paper introduces CopT (Contrastive On-Policy Thinking). It flips the script. Instead of thinking before answering, CopT acts like a detective who shouts out a draft answer immediately.

Step 1: The Gut Check. The model says, "I think the answer is X."
Step 2: The Magic Mirror. Before accepting this answer, CopT uses a special "Magic Mirror" (a contrastive mechanism) to check if the answer is trustworthy.
- It looks at the answer using a standard view (discrete tokens).
- It looks at the answer using a "blurry, uncertain" view (continuous embeddings that hold all the "maybe" possibilities the model considered).
- If the model is confident, the two views match perfectly. If the model is guessing or confused, the two views clash.
Step 3: The Decision.
- If the mirror says "Looks Good": The detective accepts the answer immediately. No need to write the long report. Result: Huge savings in time and money.
- If the mirror says "Looks Shaky": The detective says, "Okay, I need to think more." But here's the clever part: they don't just forget their first guess. They use a "visibility switch." They might hide the shaky first guess if it's confusing them, or keep it visible if it's a helpful hint, while they do the deep thinking to correct it.

The Core Metaphor: The "Blurry Lens" vs. The "Sharp Lens"

To understand the "Magic Mirror" (the contrastive verifier), imagine looking at a painting through two different lenses:

The Sharp Lens (Discrete Input): You see the final, crisp brushstrokes the model decided to paint. This is the "draft answer."
The Blurry Lens (Continuous Input): You see the entire palette of colors the model was considering before it picked the final stroke. This includes all the "what ifs" and uncertainties.

How CopT uses this:
CopT asks: "Does the Sharp Lens look the same as the Blurry Lens?"

Yes: The model was sure. The answer is likely correct. Stop thinking, save the tokens.
No: The Blurry Lens shows the model was actually very confused or considering many different possibilities, even though it picked a specific color. The answer is likely wrong. Start thinking (on-policy thinking) to fix it.

Why This Matters (According to the Paper)

The paper claims this method is a game-changer because it stops the model from "performing" unnecessary thinking.

Speed & Cost: On easy problems where the model knows the answer instantly, CopT skips the long thinking process entirely. The paper shows this cuts the "cost" (number of words/tokens generated) by nearly half in some cases.
Smarter Thinking: On hard problems where the first guess is wrong, CopT doesn't just give up. It uses the "Magic Mirror" to realize, "Wait, I'm confused," and then engages in deep thinking only when necessary.
No Training Needed: This isn't a new model that needs years of training. It's a new way of using existing models. It's like giving a smart detective a new set of rules to follow, rather than teaching them a new language.

The "Visibility Switch"

When the model realizes it needs to think more, it has to decide whether to keep looking at its first (possibly wrong) guess.

If the thinking process gets messy, CopT can hide the first guess so the model doesn't get confused by its own mistake.
If the first guess has a good hint in it, CopT shows it to help guide the correction.
This is controlled by a second "Magic Mirror" check during the thinking process.

Summary

CopT is a smarter way to use AI reasoning. Instead of forcing the AI to think for a long time before speaking, it lets the AI speak first, checks if that speech is trustworthy using a special "uncertainty detector," and only forces it to think deeply if the detector says, "Hey, that doesn't look right." This makes AI faster, cheaper, and just as smart (or smarter) on hard problems.

Technical Summary: CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning

Problem Statement

Chain-of-Thought (CoT) prompting has become the standard for eliciting reasoning capabilities in Large Language Models (LLMs). However, the prevailing CoT paradigm treats thinking as a strict prerequisite for answering, forcing models to generate a complete reasoning trace before producing a final answer. Recent observations indicate that LLMs often exhibit "performative reasoning," where they continue to generate reasoning steps even after internally identifying a plausible answer. This behavior delays access to correct answers and incurs unnecessary token costs. Furthermore, existing methods that utilize continuous embeddings (Latent CoT) often sacrifice the readability of explicit reasoning or introduce issues such as unseen representations and drifting into noise when used directly for generation.

Methodology: CopT

The authors propose CopT (Contrastive On-Policy Thinking), a training-free reasoning pipeline that reverses the standard order of operations: Answer First, Then Think. The methodology consists of two primary stages and a novel contrastive verification mechanism.

1. Reversed Reasoning Pipeline

Draft Answer Elicitation: Instead of thinking first, the model is forced to output a <answer> token immediately and generate a draft answer ( $a$ ) based on the input question ( $q$ ).
On-Policy Thinking: If the draft answer is deemed unreliable, the model triggers a subsequent thinking phase conditioned on the draft. This allows for reflection and correction only when necessary.

2. Contrastive Reliability Estimation

To determine whether a draft answer should be trusted or if further thinking is required, CopT introduces a normalized sequence-level reverse KL estimator ( $\kappa$ ). This mechanism leverages continuous embeddings as inference-time verifiers without using them for generation.

Discrete vs. Continuous Inputs: During generation, the model caches two representations for each step:
1. Discrete Prefix: The standard probability $p_t$ of the chosen token given the discrete token history.
2. Continuous Prefix: A continuous embedding $e_t$ calculated as the probability-weighted average of all vocabulary embeddings, preserving uncertainty information.
The Estimator: After generating a draft of length $T_a$ , the model compares the probability of the generated tokens under the discrete prefix ( $p_\theta(a_t | q, a_{<t})$ ) against the probability under the continuous prefix ( $p_\theta(a_t | q, e_{<t})$ ).
$\kappa_a = \frac{1}{T_a} \sum_{t=1}^{T_a} \left[ \log p_\theta(a_t | q, a_{<t}) - \log p_\theta(a_t | q, e_{<t}) \right]$
Interpretation: A high $\kappa_a$ indicates that the answer is significantly less supported when uncertainty information (continuous embeddings) is considered, suggesting the draft is unreliable. If $\kappa_a$ exceeds a threshold $\tau_a$ , the model enters the on-policy thinking stage.

3. Dynamic Visibility Control

During the on-policy thinking phase, the draft answer may still contain useful partial information but could also mislead the model if it is fundamentally flawed. CopT addresses this by dynamically controlling the visibility of the draft answer during subsequent thinking chunks.

The thinking trajectory is partitioned into chunks.
A second KL estimator ( $\kappa_r$ ) is calculated for each chunk to assess the stability of the current thinking process.
If $\kappa_r$ indicates instability (high discrepancy), the draft answer is hidden ( $m_k=0$ ) for the next chunk to prevent the model from being misled. If stable, the draft remains visible ( $m_k=1$ ).

4. Theoretical Foundation

The paper provides a theoretical analysis showing that under a "mixture-linear" assumption, the expected value of the reverse KL estimator equals the mutual information $I(S; A)$ between the unresolved latent reasoning state $S$ and the emitted answer token $A$ . This proves that the estimator measures answer-relevant uncertainty rather than arbitrary uncertainty in the latent state. Consequently, the score only increases when the uncertainty preserved in continuous embeddings actually changes the model's support for the generated tokens.

Key Contributions

Reversed Paradigm: Proposes a training-free pipeline where LLMs generate a draft answer first and invoke on-policy thinking only when the draft is deemed unreliable, enabling earlier access to answers and selective correction.
Contrastive Verification: Introduces a mechanism that recasts continuous embeddings (previously used for generation in Latent CoT) as inference-time verifiers. By contrasting discrete and continuous inputs, it estimates draft reliability and modulates draft visibility without sacrificing the readability of explicit CoT.
Theoretical Insight: Demonstrates that the proposed estimator measures mutual information between latent states and answers, explaining why it effectively captures answer-relevant uncertainty.
Empirical Validation: Extensively validates the method across mathematics, coding, and agentic reasoning tasks, showing consistent gains over standard CoT baselines.

Experimental Results

The authors evaluated CopT on Qwen3 and Qwen3.5 models (2B, 8B, and 35B scales) across 10 benchmarks, including GSM8K, Math500, AIME, HumanEval, LeetCode-Contest, and agentic benchmarks (BFCL v4, ZebraArena).

Accuracy and Efficiency: CopT improves peak accuracy by up to 23% (e.g., on ZebraArena large split) and reduces token usage by up to 57% at comparable or higher accuracy levels compared to standard CoT.
Mathematics & Coding: On GSM8K, CopT achieved a +0.23% accuracy gain with a 55.1% reduction in tokens. On challenging AIME benchmarks, it improved accuracy by +3.34% (AIME24) and +2.92% (AIME25).
Agentic Reasoning: In multi-turn agentic tasks (ZebraArena), CopT improved accuracy by +23.03% on the large split while reducing token usage by 19.6%.
Comparison with Latent Methods: Unlike training-free methods that use continuous embeddings for generation (e.g., Soft-Thinking, SwiReasoning), CopT maintains fully explicit, readable reasoning while achieving higher accuracy with fewer tokens.
Latency: The token savings translate directly to latency reductions, with CopT reducing average per-sample latency by up to 69% on HumanEval.

Significance and Claims

The paper claims that CopT offers a practical path toward more cost-effective LLM reasoning at inference time without requiring additional training. By reversing the thinking-answering order and utilizing continuous embeddings as contrastive verifiers, CopT successfully balances the trade-off between accuracy, efficiency, and reasoning transparency. The authors emphasize that this approach avoids the pitfalls of direct continuous generation (such as de-diversification and noise) while leveraging the uncertainty information inherent in continuous spaces to make smarter decisions about when to think and when to stop. The method is presented as a generalizable framework applicable to various model architectures and reasoning domains.