Technical Summary: Inject, Align, Recover (IAR)

Problem Definition

The paper addresses the challenge of document knowledge internalization in Large Language Models (LLMs). In standard Retrieval-Augmented Generation (RAG), models rely on external retrieval to answer questions based on a specific corpus. However, in many deployment scenarios (due to latency, privacy, or testing constraints), retrieval is unavailable. The goal is to convert a fixed, bounded document collection into usable parametric knowledge within the model, enabling it to answer held-out questions without access to the source documents at inference time.

Existing approaches face significant limitations:

• Supervised Fine-Tuning (SFT) on QA pairs: Provides the correct input-output format but offers sparse learning signals, as only facts selected by the QA generator contribute to the loss.
• Continued Pretraining (CPT): Offers denser exposure to document text but fails to explicitly teach the model how to answer questions under an instruction-following interface.
• The Trade-off: Both methods often suffer from catastrophic forgetting, where adapting the model to a specific domain degrades its general instruction-following capabilities and performance on broad benchmarks.

Methodology: The IAR Framework

The authors propose IAR (Inject, Align, Recover), a three-stage post-training framework designed to decouple domain acquisition, task alignment, and general capability recovery.

Stage 1: Inject

Instead of raw continued pretraining, this stage converts source documents into structured, instruction-conditioned supervised reconstruction tasks. The objective is to inject denser document-level supervision than QA-only training without using raw-stream CPT loss. The stage employs three specific objectives:

1. Continuation: Predicting a document suffix given an instruction-conditioned prefix.
2. Rewrite: Reconstructing a cleaned document from a generated summary, outline, or knowledge skeleton.
3. Instruction-Formatted Reconstruction: Predicting the cleaned document from a short, generic reading instruction.
   These objectives are mixed to create a "recipe" dataset that exposes the model to the full document structure while maintaining an instruction-following format.

Stage 2: Align

The model, now containing injected document knowledge, is fine-tuned on document-derived question-answer pairs. Crucially, this stage uses answer-only supervised fine-tuning. This aligns the injected knowledge with the QA interface, teaching the model to retrieve and utilize the internalized facts to answer specific questions. This stage produces a domain-adapted checkpoint ($\theta_{IA}$).

Stage 3: Recover

To mitigate catastrophic forgetting, the framework performs post-hoc model merging. It merges the domain-adapted checkpoint ($\theta_{IA}$) with the original base instruction model ($\theta_0$). The authors evaluate several merging operators, including SLERP, task arithmetic, TIES, and DARE.

• Selection Strategy: The final checkpoint is not necessarily the one with the highest domain accuracy. Instead, the authors select a point on the domain-general frontier. They prioritize domain accuracy as the primary objective while using general benchmarks (IFEval, MMLU, MSBench) as "guardrails" to ensure deployability.

Key Contributions

1. Problem Formulation: The paper frames retrieval-free QA over a fixed corpus as a domain-general operating-point problem, explicitly measuring the trade-off between domain accessibility and general capability.
2. IAR Framework: It introduces a three-stage decomposition (Inject, Align, Recover) that separates structured document exposure, answer-only alignment, and capability recovery. This allows for the isolation of which intervention drives specific gains or failures.
3. Comprehensive Evaluation: The method is evaluated across two corpora (Common Corpus/CC and CCI) and four model families (Llama, Phi, Qwen, SmolLM). It compares against a wide range of baselines, including Vanilla SFT, CPT+SFT, LoRA, SDFT, Replay, and FAPM.
4. Empirical Evidence: The study provides both strong and boundary evidence, clarifying when initialization strength, data recipes, or operating-point selection are most critical.

Results

The experiments demonstrate that IAR improves the domain-general frontier for retrieval-free document internalization:

• Performance vs. Vanilla SFT: IAR improves over Vanilla SFT on all four reported metrics in 7 of 8 dataset-model settings.
  • Exception: In the Phi-4-mini CCI setting, IAR improved IFEval and MSBench but slightly reduced domain accuracy and MMLU relative to Vanilla SFT.
  • Average Gains: Across the settings, IAR achieves an average increase of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance (across IFEval, MMLU, and MSBench) compared to Vanilla SFT.
  • Specific Example: On the CC dataset with Qwen3-4B, IAR improved domain accuracy from 42.4% (Vanilla SFT) to 50.5% while simultaneously improving all three general metrics.
• Comparison with BudgetMatch: When compared to a "BudgetMatch" baseline (which uses the same token budget but only QA-only training), IAR showed superior performance in three of four settings, indicating that the gains are not solely due to increased token count but also the staged exposure and recovery.
• Scaling: On the CC dataset, scaling Qwen3 models (8B, 14B, 32B) showed that IAR maintains domain accuracy within 1.1 points of the best pre-recovery checkpoint while recovering 14.9–24.1 points in mean general performance.
• Boundary Cases: The paper notes that IAR does not uniformly dominate every metric in every setting (e.g., Phi-4-mini on CCI showed slight domain accuracy reductions but general metric improvements), highlighting that deployment preferences dictate the optimal operating point.

Significance and Claims

The paper claims that IAR improves the operating-point frontier for retrieval-free document internalization. Its significance lies in demonstrating that:

1. Decomposition is Effective: Separating document exposure, QA alignment, and recovery is more effective than collapsing these functions into a single fine-tuning recipe.
2. Recovery is Critical: Post-hoc merging is a viable strategy to recover general capabilities lost during domain adaptation without sacrificing the internalized domain knowledge.
3. Measurable Trade-offs: Document exposure, answer alignment, data recipes, and recovery should be measured separately. The paper argues against collapsing these into a single score, as different baselines (like LoRA or FAPM) may win on specific general metrics but fail on domain internalization, whereas IAR offers a balanced, strong domain-primary operating point.

The authors conclude that IAR represents a strong, setting-dependent operating point rather than a universally dominant recipe, providing a robust framework for internalizing document knowledge while preserving general model utility.