Technical Summary: OCC-RAG: Optimal Cognitive Core for Faithful Question Answering

Problem Statement

Recent advancements in Large Language Models (LLMs) have been driven primarily by scaling, resulting in models that absorb vast amounts of world knowledge into their weights. However, many practical applications, particularly those requiring Context Question Answering (Context QA), benefit more from robust reasoning and strict adherence to provided evidence than from extensive parametric knowledge.

Current LLMs face three critical challenges in Context QA:

1. Lack of Faithfulness: Models often prioritize their internal parametric knowledge over the provided context, leading to hallucinations or answers that contradict the input text (e.g., answering based on real-world facts when the context presents a counterfactual scenario).
2. Insufficient Multi-hop Reasoning: Models struggle to synthesize information across disparate parts of a context to bridge logical gaps without relying on memorized shortcuts.
3. Inability to Abstain: Models frequently fail to recognize when a context is insufficient, ambiguous, or lacks necessary information, leading to confident but incorrect answers rather than a calibrated refusal.

While Small Language Models (SLMs) offer a principled, cost-effective alternative, they often lack the specialized training to perform these tasks reliably without hallucinating or defaulting to memorized knowledge.

Methodology

The authors introduce Optimal Cognitive Core (OCC), a family of SLMs designed to prioritize task-specific reasoning capabilities over knowledge capacity. The specific variant presented, OCC-RAG, is optimized for faithful, context-grounded question answering.

1. Model Architecture and Training Strategy

• Base Models: The authors selected Qwen3-0.6B-Base and Qwen3-1.7B-Base as the starting points, determined to be the strongest performers among open-weight SLM families (including Gemma3 and SmolLM3) on initial QA slices.
• Mid-Training: Instead of pre-training from scratch, the models undergo mid-training (supervised fine-tuning) on a massive synthetic corpus. This stage is designed to explicitly shape the model's reasoning architecture for Context QA.
• Training Objective: The loss is applied only to response tokens. The model is trained to generate structured reasoning traces that include source citations grounded in literal quotes from the context.

2. Synthetic Data Pipeline

The core of the methodology is a novel pipeline for synthesizing a corpus of over 3.25 million QA examples. The data is designed to enforce three specific behaviors: multi-hop inference, strict context faithfulness, and calibrated abstention.

• Data Composition:
  • Single-hop QA: The largest subset (~2.78M pairs), generated by ingesting Wikipedia pages, cleaning text, and generating extractive QA pairs.
  • Multi-hop QA: Generated using a Knowledge Graph (KG) extracted from the context (gold and distractor chunks) via the Wikontic pipeline. This ensures the existence of "bridge" entities and provides structural control over reasoning complexity. The pipeline samples paths from the KG to generate questions requiring information fusion across multiple paragraphs.
  • Unanswerable (Abstention) QA: A subset of ~43k examples where the context is deliberately insufficient. These are constructed by using a fine-tuned DeBERTa model to identify cases where an extractive model cannot recover the answer from a reduced context, signaling missing information.
• Structured Reasoning Traces: Every training instance includes an explicit reasoning trace formatted into four sections:
  1. Query Analysis: Identifies the question's intent and entities.
  2. Source Analysis: Attributes facts to specific source passages.
  3. Reasoning: Combines facts to derive the answer.
  4. Status and Answer: An explicit ANSWERABLE/UNANSWERABLE verdict followed by the final answer.
• Data Mixing Strategy: To ensure the model develops reasoning capabilities, the training curriculum oversamples multi-hop examples (shown 3 times per epoch) compared to single-hop examples (shown once), despite single-hop data dominating the token budget.

3. Evaluation Benchmarks

The models are evaluated on three core dimensions:

• Multi-hop Reasoning: HotpotQA, MuSiQue, and TAT-QA (measuring In-Accuracy and F1).
• Faithfulness: ConFiQA, which tests performance on counterfactual contexts. Metrics include In-Accuracy and the Memorization Ratio (MR), measuring how often the model defaults to parametric knowledge over context.
• Refusal: MuSiQue-Un, measuring the ability to correctly identify unanswerable questions (Refusal Accuracy).

Key Results

The released models, OCC-RAG-0.6B and OCC-RAG-1.7B, demonstrate that compact, task-specialized SLMs can match or exceed general-purpose models 2–6 times their size.

• Performance vs. Size:
  • OCC-RAG-0.6B outperforms Gemma3-4B and SmolLM3-3B across all dimensions (multi-hop, faithfulness, and refusal).
  • OCC-RAG-1.7B achieves competitive or superior results compared to Qwen3-4B in thinking mode on multi-hop reasoning and surpasses it significantly on faithfulness and refusal.
• Faithfulness and Memorization:
  • OCC-RAG-1.7B reduces the Memorization Ratio (MR) from 12.7 (Qwen3-1.7B) to 5.0, demonstrating a strong ability to prioritize provided context over internal knowledge.
  • On ConFiQA, OCC-RAG-0.6B exceeds Qwen3-1.7B by 9.5 points.
• Refusal Capabilities:
  • OCC-RAG-1.7B achieves 87.2% Refusal Accuracy, performing on par with models of 8B parameters or higher, whereas smaller baselines struggle significantly.
• Comparison to Specialized Baselines:
  • OCC-RAG-0.6B surpasses the prior specialized baseline, Pleias-RAG-1.2B, by 21.6 points on MuSiQue and significantly outperforms it on faithfulness metrics.

Significance and Claims

The paper posits that faithfulness does not require scale alone. Instead, it can be learned through a specific training curriculum and supervision format.

• Optimal Cognitive Core: The authors argue that for applications where correctness must be tied to evidence (e.g., legal, medical, or fact-checking domains), an "optimal cognitive core"—a compact, task-specialized architecture—is a practical and efficient alternative to ever-larger general-purpose models.
• Learned Abstention: The work demonstrates that "safe abstention" can be transformed from a heuristic into a learned reasoning behavior through structured supervision, making SLMs more reliable in high-stakes settings.
• Efficiency: By mid-training on a synthetic corpus of structured reasoning traces, the authors show that SLMs can internalize the process of multi-hop inference and evidence grounding without the computational cost of scaling up model parameters or using full "thinking mode" inference.

The paper concludes that OCC-RAG provides a reusable recipe for building transparent, efficient, and robust QA systems that are resilient to missing or conflicting evidence.