RASPRef: Retrieval-Augmented Self-Supervised Prompt Refinement for Large Reasoning Models

1. Problem Statement

Recent Large Reasoning Models (LRMs) like DeepSeek R1 and OpenAI o1 have demonstrated strong capabilities in structured reasoning tasks (e.g., mathematics, code generation, multi-hop QA). However, their performance is highly sensitive to prompt formulation.

• Current Limitation: Designing effective prompts is typically a manual, iterative process that does not scale well across different tasks or domains.
• Gap in Existing Solutions:
  • Traditional prompt optimization often requires labeled data, reward models, or access to model weights (gradients), making them unsuitable for closed-source API-based models.
  • Existing self-improvement methods focus on refining the model's output rather than optimizing the prompt itself.
• Goal: Develop a framework that automatically refines prompts without human annotations, task-specific supervision, or access to internal model parameters, leveraging the model's own accumulated reasoning history.

2. Methodology: RASPRef Framework

RASPRef (Retrieval-Augmented Self-Supervised Prompt Refinement) is a black-box, iterative framework that treats the prompt as an adaptive component. It combines Retrieval-Augmented Generation (RAG) with Self-Supervised Evaluation to optimize prompts during inference.

Core Components

The framework operates on a store of past reasoning trajectories, where each entry includes:

1. Input problem.
2. The prompt used.
3. The generated reasoning trace.
4. Self-supervised signals (consistency, verifier scores, critiques).

The Optimization Process (4 Steps)

1. Retrieval of Relevant Trajectories: For a new input $x$, the system retrieves the top-$k$ most similar past trajectories from the memory store based on task structure or domain characteristics.
2. Initial Prompt Construction: An initial prompt is built using:
   • General task instructions.
   • Guidelines distilled from high-quality retrieved traces.
   • Representative examples from the retrieved set.
3. Self-Supervised Evaluation: The model generates $K$ reasoning traces using the candidate prompt. These are evaluated using a composite quality score $Q(p; x)$ that aggregates four signals (no ground-truth labels required):
   • Consistency ($C_{cons}$): Agreement across multiple sampled answers.
   • Verifier Score ($C_{ver}$): Plausibility and structural correctness judged by a lightweight verifier model.
   • Self-Critique ($C_{crit}$): The model's own analysis of its reasoning errors.
   • Retrieval Alignment ($C_{ret}$): How well the reasoning reuses elements from the retrieved trajectories.
4. Iterative Refinement: The model acts as an editor, reviewing the prompt and the generated traces to propose targeted revisions (e.g., "clarify step structure," "avoid premature answers"). This loop repeats until the prompt stabilizes or a computational budget is reached.

Mathematical Formulation

The goal is to find a prompt $p^*$ that maximizes the expected quality score:
$$p^* \approx \arg \max_{p \in \mathcal{P}} \mathbb{E}_{x \sim D_{test}} [Q(p; x)]$$
Where $Q(p; x) = \alpha C_{cons} + \beta C_{ver} + \gamma C_{crit} + \delta C_{ret}$.

3. Key Contributions

• Prompt-Centric Optimization: Unlike prior work that optimizes model outputs, RASPRef explicitly treats the prompt as the optimization target, viewing it as an adaptive component that evolves with accumulated experience.
• Label-Free Self-Supervision: The framework operates entirely without human labels or task-specific reward models, relying instead on internal consistency, verifier models, and self-critique.
• Retrieval-Guided Adaptation: It bridges the gap between RAG and prompt engineering by using past successful reasoning trajectories to construct and refine prompts for new, similar problems.
• Black-Box Compatibility: The method works with frozen, API-accessible models (e.g., GPT-4o-mini) without requiring gradient access or fine-tuning.

4. Experimental Results

The authors evaluated a prototype of RASPRef on the GSM8K benchmark (grade-school math word problems requiring multi-step reasoning).

• Setup:
  • Model: GPT-4o-mini (frozen, accessed via API).
  • Data: 500 test examples; retrieval index built from 200 training examples.
  • Baselines: Static Chain-of-Thought (CoT) prompting vs. Retrieval-Augmented Prompting.
• Findings:
  • Static Prompting Accuracy: 85.6%
  • Retrieval-Augmented Prompting Accuracy: 95.0%
• Analysis: The retrieval of relevant reasoning trajectories provided crucial contextual guidance, helping the model structure intermediate steps more effectively and reducing reasoning errors. The results validate that retrieval-guided prompt construction significantly outperforms static baselines.

5. Significance and Future Directions

• Scalability: RASPRef offers a practical, scalable strategy for improving reasoning performance in large language models without the cost of fine-tuning or human annotation.
• Interpretability: The refined prompts remain transparent, revealing the heuristics and patterns extracted from successful reasoning traces, unlike black-box parameter tuning.
• Robustness: The framework shows potential for generalizing to long-tail or rare cases by retrieving deep reasoning patterns rather than just superficial lexical similarities.
• Future Work: The authors suggest expanding to broader domains (e.g., multi-hop QA, program synthesis), integrating stronger verification signals, and optimizing the efficiency of the refinement loop.

In conclusion, RASPRef demonstrates that prompt design is a critical, optimizable factor for reasoning models and that self-improving, retrieval-augmented prompts can significantly enhance performance in a label-free setting.