Technical Summary: CORE – Contrastive Reflection Enables Rapid Improvements in Reasoning

Problem Statement

Language models possess the capacity to improve reasoning tasks using verifiable rewards, yet current approaches suffer from significant inefficiencies. Parametric methods (e.g., GRPO) and non-parametric methods (e.g., GEPA, prompt optimization) typically require hundreds of training samples and thousands of model rollouts to achieve meaningful progress. This makes them computationally expensive and often intractable for rapid adaptation. In contrast, humans can often master new tasks with only a handful of practice problems. The central challenge addressed by this work is how to enable language models to learn from verifiable rewards with human-like sample and rollout efficiency, while maintaining interpretability and avoiding the opacity of weight updates.

Methodology: CORE (Contrastive Reflection)

CORE is a non-parametric learning algorithm designed for a frozen language model. Instead of updating model weights or optimizing a single global prompt, CORE learns by generating and accumulating "insights"—short, natural-language descriptions of reasoning strategies or constraints that distinguish successful problem attempts from unsuccessful ones.

The algorithm operates through four primary components:

1. External Memory Stores:

   • Rollout Memory ($R$): Stores past reasoning traces (problem, retrieved insights, model output, reward) indexed by problem embeddings.
   • Insight Memory ($I$): Stores natural-language insights paired with empirical utility estimates. For each insight, the system tracks the number of applications and the mean baseline-relative utility.

2. Failure-Biased Problem Sampling:
   To maximize learning efficiency, CORE biases sampling toward problems the model solves less reliably. It calculates a sampling weight based on the current accuracy estimate, mixing failure-biased sampling with uniform sampling to ensure coverage of the training set.

3. Contrastive Reflection (The Core Mechanism):
   When the model fails to solve a problem, CORE triggers a reflection step. It retrieves a successful rollout ($\tau^+$) from a semantically similar problem (or the same problem if previously solved) and contrasts it with the failed rollout ($\tau^-$). The frozen model is prompted to generate candidate insights describing the strategies or constraints that explain the difference between success and failure.

   • Admission Testing: Candidate insights are not immediately stored. They undergo an admission test where the model attempts the originating problem using only that candidate insight. An insight is admitted to memory only if it improves the success rate over the baseline (specifically, if it leads to a correct solution on the first attempt in the experiments).

4. Utility-Aware Retrieval:
   During inference, CORE retrieves insights based on a score combining semantic similarity (between the current problem and past problems where the insight was used) and empirical utility (the observed success rate of the insight on similar problems). This allows the model to selectively apply insights that have proven effective in the past.

Key Contributions

1. Algorithmic Innovation: Introduction of CORE, a non-parametric algorithm that enables frozen language models to learn from verifiable rewards by generating, testing, and accumulating abstract natural-language insights rather than updating weights or optimizing prompts.
2. Efficiency Gains: Demonstration across four reasoning tasks (Tower of Hanoi, MathGAP, ZebraLogic, Matchstick Arithmetic) that CORE outperforms both parametric (GRPO) and non-parametric (GEPA, Episodic RAG, MemRL) baselines in sample efficiency (learning with fewer training samples) and rollout efficiency (learning with fewer model attempts).
3. Context Efficiency: Evidence that CORE is substantially more context-efficient than non-parametric baselines. While methods like Episodic RAG and MemRL add tens of thousands of tokens to the prompt by retrieving full traces, CORE compresses experience into compact insights, adding significantly fewer tokens per evaluation.
4. Interpretability: The learned artifacts are transparent natural-language insights paired with utility estimates, offering a more interpretable alternative to opaque parameter updates or prompt optimization.
5. Ablation Analysis: Identification that both contrastive reflection (comparing failure vs. success) and utility-aware retrieval are necessary components for the algorithm's performance.

Experimental Results

The authors evaluated CORE on four reasoning tasks using the gpt-oss-120b model, comparing it against GRPO, GEPA, Episodic RAG, and MemRL across training regimes of 5, 10, and 100 samples.

• Rollout Efficiency: In the 10-sample regime, CORE achieved higher held-out accuracy than any baseline by 350 training rollouts. Even when baselines were allowed 4,000 rollouts and CORE was limited to 2,100, CORE settled on higher final performance.
• Sample Efficiency: Across 12 task-by-data-regime conditions, CORE achieved the highest mean held-out accuracy in 9 of them. It improved over the no-learning baseline by approximately 54–56% across 5, 10, and 100 training examples.
• Context Efficiency: CORE added an average of 0.92k tokens per evaluation item, compared to ~33k for Episodic RAG and ~32k for MemRL. It was approximately 1.4x more context-efficient than GEPA.
• Insight Quality: Analysis of admitted insights revealed they fell into three functional categories: Search space structuring (heuristics to reduce possibilities), Intermediate state tracking (updating quantities/moves), and Verification and validation (checking constraints). These insights were found to be procedural abstractions rather than summaries of specific episodes.

Significance and Claims

The paper posits that the efficiency of human learning stems from the ability to revisit past successes and failures to discover abstract, explicit principles. CORE operationalizes this by distilling contrasts between successful and unsuccessful traces into reusable knowledge.

The authors claim that CORE offers a more efficient and interpretable route to model self-improvement than weight updates, prompt optimization, or direct reuse of stored reasoning traces. By storing "credit-assignment hypotheses" (insights) rather than raw data or weights, CORE reduces the risk of introducing unwanted behaviors associated with opaque updates. The work suggests that the key to rapid improvement lies not just in what is learned, but in how it is represented and retrieved: as compact, utility-weighted natural language insights derived from contrastive reflection.

Limitations noted by the authors:

• CORE assumes access to verifiable rewards, limiting applicability to domains with clear binary feedback.
• The utility update assigns the same outcome to all retrieved insights, leaving finer-grained credit assignment among multiple insights unresolved.
• Reflection and admission testing introduce additional inference costs.
• Experiments are currently limited to single-turn reasoning, planning, and problem-solving tasks.