GATES: Self-Distillation under Privileged Context with Consensus Gating

1. Problem Statement

The paper addresses the challenge of self-improving language models in the absence of ground-truth labels, verifiable rewards, or external graders.

• The Setting: The authors focus on Document-Grounded Question Answering (DGQA) with asymmetric context.
  • Tutor Role: The model has access to a relevant source document ($d$) and a question ($q$) during training. This privileged context allows for more reliable reasoning.
  • Student Role: The same model (sharing weights) must answer the question ($q$) without the document at test time.
• The Challenge: Standard self-distillation often fails because the "teacher" is the model itself. Without external verification, naive self-distillation can reinforce systematic errors or encourage degenerate shortcuts (e.g., the model learning to rely on document references it won't have later). The core difficulty is distinguishing between reliable supervision (where the model's reasoning is correct) and unreliable supervision (where the model is hallucinating or making errors) without ground truth.

2. Methodology: GATES

The authors propose GATES (Gated Asymmetric Trajectory Self-distillation), a framework that uses consensus gating to filter training signals before distilling knowledge.

Core Mechanism: Consensus Gating

Instead of assuming the tutor is correct, GATES derives supervision reliability online:

1. Sampling: For each training question, the tutor (with document access) generates $k$ independent reasoning rollouts.
2. Consensus Check: The system extracts the final answer from each rollout. If at least $\tau$ out of $k$ rollouts agree on the same answer, the question is marked as reliable (consensus gate = 1).
3. Filtering:
   • If consensus is weak, the question is skipped entirely (zero loss).
   • If consensus is strong, the specific tutor trajectories that match the consensus answer are selected for distillation.
   • Guardrails: A keyword filter ensures tutor trajectories do not explicitly reference the document (preventing "document leakage" where the student learns to cite a source it won't see).

Training Objectives

GATES employs two complementary distillation modes, both gated by the consensus signal:

1. Off-Policy Distillation (Primary Driver): The student model is trained to imitate the full reasoning trajectories (Chain-of-Thought + Final Answer) of the consensus-approved tutor rollouts. This provides dense, token-level supervision.
   • Loss: Negative Log-Likelihood (NLL) on tutor tokens, normalized by the number of valid tokens.
2. On-Policy Distillation (Secondary): The student generates its own trajectories. The tutor (with document access) scores these tokens. An advantage signal ($A_t$) upweights tokens where the document-aware tutor assigns a higher probability than the student.
   • Loss: Advantage-weighted NLL.

Key Distinction: Unlike standard self-consistency (which selects answers at inference), GATES uses consensus to gate the training updates. It decides when to learn, not just what to answer.

3. Key Contributions

1. Formalization of Asymmetric-Context Self-Distillation: The paper defines a setting where a single model acts as both tutor and student under different input contexts (privileged vs. non-privileged), enabling self-improvement without external labels.
2. Introduction of GATES: A novel method that uses agreement among multiple document-grounded tutor responses to gate which reasoning trajectories are distilled.
3. Empirical Validation of Consensus Gating: The authors demonstrate that the consensus gate is the critical mechanism. Ablation studies show that removing the gate causes performance to collapse to the level of unfiltered trajectory distillation, while adding sparse correctness rewards provides no additional benefit.

4. Experimental Results

Experiments were conducted using Qwen3-4B-Base as the underlying model, trained on a dataset of 551 math questions derived from the Nemotron-CC-Math corpus.

• In-Domain Performance (Held-out Evaluation):

  • GATES: Improved student accuracy from 46.0% to 62.0%.
  • Baselines:
    • Answer-Only SFT: Catastrophically degraded accuracy to ~10–12%.
    • Outcome RL (Sparse Reward): No meaningful improvement over the base model (~21%).
    • Tutor-Trajectory SFT (No Gating): Achieved 54.0%, confirming that filtering unreliable data is crucial.

• Out-of-Domain Performance (Public Math Benchmarks):

  • Evaluated on MATH, AMC, Minerva, and OlympiadBench using majority voting (maj@8).
  • GATES: Improved average accuracy from 20.2% (Base Model) to 35.4%.
  • Comparison: Outperformed the next best baseline (Tutor-Trajectory SFT at 32.3%) by 3.1 percentage points.

• Ablation Findings:

  • Removing the consensus gate resulted in a significant performance drop, proving that reliability filtering is the "active ingredient."
  • Adding a sparse "oracle" correctness reward (based on tutor agreement) did not improve performance and sometimes hurt in-domain accuracy, suggesting dense trajectory imitation is sufficient when gated by consensus.

5. Significance and Implications

• Solving the "No-Label" Problem: GATES demonstrates that models can self-improve in high-stakes reasoning tasks without human annotations or external reward models, provided there is a mechanism to verify internal consistency.
• Efficiency of Dense Supervision: The results highlight that dense token-level supervision (imitating full reasoning paths) is far superior to sparse reward signals (Outcome RL) or answer-only fine-tuning, provided the supervision is reliable.
• Robustness to Self-Error: By filtering out questions where the model cannot reach a consensus, GATES prevents the "error amplification" common in self-training loops.
• Generalizability: While tested on math, the framework applies to any domain where privileged information (e.g., search results, tool outputs, environment state) is available at training time but not at test time (Retrieval-Augmented Generation, Agentic workflows).

In conclusion, GATES establishes that consensus gating is a minimal yet sufficient mechanism to make self-distillation robust under unreliable supervision, enabling significant gains in reasoning capabilities without external ground truth.