💬 NLP

ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure

ConPress is a lightweight self-supervised fine-tuning method that leverages the "Self-Compression" phenomenon, where models naturally shorten reasoning traces when presented with multiple questions, to train large reasoning models to generate concise and accurate solutions for single-question tasks, significantly reducing inference overhead without external teachers or reinforcement learning.

Original authors: Jie Deng, Shining Liang, Jun Li, Hongzhi Li, Yutao Xie

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jie Deng, Shining Liang, Jun Li, Hongzhi Li, Yutao Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, over-enthusiastic student who is incredibly smart but has a bad habit: overthinking.

When you ask this student a simple math question like "What is 2 + 2?", they don't just say "4." Instead, they write a 50-page essay. They start by wondering if they remember addition correctly, then they visualize apples, then they check their fingers, then they debate the philosophical nature of numbers, and finally, after 20 pages of "wait, let me double-check," they arrive at the answer.

This is exactly what modern "Large Reasoning Models" (AI) do. They generate massive amounts of text (called "Chain-of-Thought") to solve problems. While this often leads to correct answers, it's incredibly slow, expensive, and full of fluff.

The paper ConPress introduces a clever trick to teach this student to be concise without losing their intelligence. Here is how it works, broken down into simple concepts:

1. The Discovery: The "Crowded Room" Effect

The researchers noticed something funny happening when they changed the rules of the game.

The Old Way (Single Question): If you ask the AI one question in a quiet room, it feels like it has all the time in the world. It takes its time, rambling through every possible thought path.
The New Way (Multi-Question Pressure): If you put three or four different questions in the same prompt and tell the AI, "Answer all of these right now," something magical happens. The AI suddenly stops rambling.

The Analogy: Imagine you are at a dinner party.

If you are the only one talking to the host, you might tell a long, winding story with lots of details.
But if the host says, "Okay, we have 10 people here, and we need to hear a quick story from everyone before dessert," you suddenly get to the point. You stop the "umms," the "let me think about this," and the "wait, let me check that." You just give the core story.

The researchers call this "Self-Compression." The pressure of having to answer multiple questions at once forces the AI to cut out the fluff and get straight to the logic.

2. The Solution: ConPress (Contextual Pressure)

The team realized they could use this "crowded room" effect to train the AI to be efficient permanently. They created a method called ConPress.

Here is the step-by-step process they used:

The Stress Test: They took a bunch of math problems and grouped them into batches (e.g., 3 questions at a time). They asked the AI to solve all of them in one go.
The Filter: Because the AI was rushing, it sometimes made mistakes. So, the researchers only kept the answers that were 100% correct. They threw away the wrong ones.
The Lesson: They took those short, correct answers (the "compressed" reasoning) and used them to teach the AI how to answer single questions in the future.

The Analogy: It's like a coach who watches a sprinter run a race while carrying a heavy backpack (the multi-question pressure). The sprinter learns to run efficiently to carry the weight. The coach then takes the sprinter, removes the backpack, and says, "Now, run like you did when you had the backpack." The sprinter keeps the efficient stride even without the weight.

3. The Results: Faster, Cheaper, Still Smart

The results were impressive. By using this method, the AI models became significantly more efficient:

Less Talking: The models used 30% to 60% fewer words (tokens) to solve the same problems.
Same Smarts: Despite saying much less, they still got the right answers almost as often as before.
No External Teachers: Unlike other methods that require a "teacher" AI to rewrite the answers or complex reward systems, this method taught the AI using its own behavior. It was a "self-supervised" lesson.

What They Did NOT Claim

It is important to stick to what the paper actually says:

They did not claim this works for medical diagnosis or legal advice.
They did not claim this makes the AI "smarter" in terms of raw intelligence; it just makes it faster and cheaper at using the intelligence it already has.
They did not suggest this works by simply telling the AI "be short" at the moment of answering. The paper explicitly tested that and found it didn't work well. The AI has to learn the habit through training (ConPress) to keep the efficiency.

Summary

The paper ConPress discovered that if you squeeze a reasoning AI by asking it multiple questions at once, it naturally stops overthinking and starts being concise. They used this "squeeze" to train the AI to be efficient even when asked just one question later. The result is a smarter, faster, and more cost-effective AI that doesn't waste time rambling.

Technical Summary: ConPress – Learning Efficient Reasoning from Multi-Question Contextual Pressure

Problem Statement

Large Reasoning Models (LRMs), such as OpenAI-O1, DeepSeek-R1, and Qwen3, achieve high performance on complex tasks by generating extensive Chain-of-Thought (CoT) traces. However, this verbosity often leads to "overthinking," where models produce redundant intermediate steps, significantly increasing inference latency, token costs, and deployment overhead. Existing methods to mitigate this, such as Supervised Fine-Tuning (SFT) with teacher models or Reinforcement Learning (RL) with length penalties, suffer from high training costs, reliance on external supervision, or sensitivity to reward design. The paper addresses the fundamental question of whether LRMs can naturally produce shorter reasoning traces and how to internalize this behavior for efficient single-question inference without external teachers or RL.

Methodology: ConPress

The authors propose ConPress (Learning from Contextual Pressure), a lightweight, self-supervised fine-tuning framework. The method is built upon the observation of a reproducible inference-time phenomenon termed Self-Compression.

1. The Self-Compression Phenomenon

The authors identify that when an LRM is presented with a prompt containing multiple independent, answerable questions ( $N > 1$ ), it spontaneously generates significantly shorter reasoning traces for each individual question compared to single-question prompts ( $N=1$ ).

Mechanism: This compression arises from "multi-question contextual pressure." The model must complete multiple reasoning processes within a shared response, biasing local continuations toward concise paths that remove redundant steps while preserving core inference structures.
Characteristics: The phenomenon is reproducible across different models (e.g., DeepSeek-R1-Distill-Qwen, Qwen3) and datasets. Compression intensity scales with the number of questions ( $N$ ) and is largely structural, remaining robust even if the auxiliary questions are trivial.

2. The ConPress Pipeline

ConPress leverages this phenomenon to distill efficient reasoning into standard single-question settings through a three-step process:

Multi-Question Sampling: The model samples $N$ independent questions from a dataset and packs them into a single prompt. The model generates reasoning traces and answers for all questions within this context.
Compressed Trace Extraction: The authors apply rejection sampling, retaining only the reasoning traces ( $r_i$ ) where the predicted answer ( $\hat{o}_i$ ) matches the ground truth ( $o_i$ ). This ensures that the distilled data consists of concise yet correct trajectories.
Single-Question Transfer: The filtered dataset of concise, correct traces is used to perform standard Supervised Fine-Tuning (SFT) on single-question inputs. The model learns to reproduce the compressed reasoning behavior observed under multi-question pressure, internalizing it for standard inference.

Crucially, this process requires no external teacher models, manual pruning heuristics, or reinforcement learning.

Key Contributions

Identification of Self-Compression: The paper systematically characterizes a reproducible inference-time phenomenon where LRMs generate shorter per-question CoT traces under multi-question contexts without explicit length constraints.
ConPress Framework: A lightweight, self-supervised method that extracts and transfers this compressed reasoning behavior from multi-question to single-question settings.
Empirical Validation: The authors demonstrate that ConPress achieves substantial reductions in reasoning token usage (30–60%) on challenging benchmarks while maintaining competitive accuracy, exposing a favorable accuracy-efficiency trade-off.

Experimental Results

The authors evaluated ConPress on three reasoning models (Qwen3-4B-Thinking, R1-Distill-Qwen-7B, R1-Distill-Qwen-1.5B) across multiple benchmarks (MATH500, AIME25, GSM8K, Olympiad, AMC).

Token Reduction: ConPress reduced reasoning token usage by 59% on MATH500 and 33% on AIME25 for the Qwen3-4B-Thinking model. Across all models, average token reductions ranged from 33.9% to 48.7%.
Accuracy Preservation: Despite aggressive compression, accuracy degradation was minimal. For Qwen3-4B-Thinking, the average accuracy drop was only 0.6 points; for R1-Distill-Qwen-7B, accuracy was effectively preserved.
Comparison with Baselines:
- RFTshortest (selecting the shortest correct response from multiple samples) yielded only marginal token reductions.
- DPOshortest achieved moderate compression but often at the cost of noticeable accuracy drops.
- RL-based methods (e.g., LC-R1, AdaptThink) achieved strong compression but required complex training pipelines and suffered larger accuracy declines.
- Test-time Compression (directly using multi-question prompts at inference) reduced tokens but caused severe accuracy degradation (e.g., -31.3 points on average for MQ-N3).
Generalization: The method generalized to out-of-distribution tasks (MMLU-STEM), reducing tokens by ~33–36% with minor accuracy loss.
Ablation Studies: Increasing the number of questions ( $N$ ) in the sampling prompt increased compression strength but with diminishing returns on token savings beyond $N=3$ or $N=4$ . The method was robust to the position of the target question within the multi-question prompt.

Significance and Claims

The paper claims that ConPress demonstrates that self-compressed reasoning behaviors induced at inference time can be systematically internalized by the model itself. By leveraging the structural pressure of multi-question contexts, the method enables more token-efficient reasoning without relying on explicit length constraints, external teachers, manual pruning, or reinforcement learning.

The authors position this as a fundamental shift in how efficiency can be achieved: rather than forcing models to be concise through external penalties or complex RL, one can exploit the model's inherent contextual adaptability to "teach" it to reason efficiently. The approach is presented as a simple, scalable, and cost-effective alternative to existing compression strategies, offering a clear trade-off between reasoning length and accuracy that favors efficiency with minimal performance loss.