Technical Summary: Compute as Teacher (CaT)

Problem Statement

Post-training large language models (LLMs) for specialized skills typically relies on Supervised Fine-Tuning (SFT) with human-labeled reference answers or Reinforcement Learning (RL) with verifiable rewards from programmatic checkers. While effective for domains like mathematics and code, these approaches fail in non-verifiable domains (e.g., healthcare guidance, creative writing, freeform dialogue) where:

1. Multiple valid answers exist, and experts may disagree.
2. Deterministic rule-checking is impractical.
3. Human annotation is expensive, hard to scale, or unavailable.

Existing alternatives, such as using an LLM as a judge to score freeform outputs, suffer from inconsistency, verbosity bias, and reward hacking. The paper asks: Can inference-time compute substitute for missing supervision?

Methodology: Compute as Teacher (CaT)

The authors propose Compute as Teacher (CaT), a framework that converts inference-time compute into supervision signals for RL training without requiring ground-truth labels. The framework operates in two primary stages, leveraging parallel rollouts generated by the current policy $\pi_t$.

1. Reference Estimation

Instead of relying on a human-provided reference, CaT aggregates $G$ parallel rollouts ($o_{1:G}$) generated for a prompt $q$ into a pseudo-reference ($s$). This aggregation captures the model's uncertainty, as rollouts often differ precisely where the model is unsure.

• Aggregation Strategies: The framework is agnostic to the method. It explores Synthesis, where a frozen anchor policy ($\pi_0$) conditions on the set of rollouts to generate a new, reconciled response. This allows the system to combine correct fragments from different rollouts (e.g., correct intermediate steps from one and a valid conclusion from another).
• Alternatives: Selection strategies like Majority Vote or Best-of-N are also compatible, though synthesis is highlighted for its ability to construct responses outside the original rollout distribution.

2. Reward Derivation

The pseudo-reference $s$ is used to derive rewards for the individual rollouts to update the policy via Group Relative Policy Optimization (GRPO). The mechanism differs based on domain verifiability:

• Verifiable Domains (e.g., Math, Code): Rewards are binary, based on whether a rollout's final answer matches the pseudo-reference's answer.
• Non-Verifiable Domains (e.g., Healthcare): The paper introduces Self-Proposed Rubrics.
  • The anchor policy $\pi_0$ generates a set of binary, verifiable criteria ($R = \{r_1, \dots, r_n\}$) derived specifically from the pseudo-reference $s$ (e.g., "recommends consulting a medical professional").
  • An independent judge model ($\pi_J$) scores each rollout against these criteria.
  • The reward is the fraction of criteria satisfied. This decomposes fuzzy qualitative judgments into specific, auditable binary checks, mitigating the instability of direct LLM-as-a-judge scoring.

Training Loop

The process iterates: the policy generates rollouts $\to$ reference estimation creates $s$ $\to$ reward derivation scores rollouts against $s$ $\to$ GRPO updates the policy. Crucially, the framework amortizes the compute cost of generating multiple rollouts and aggregating them during training, allowing the final trained policy to produce high-quality single outputs at inference time.

Key Contributions

1. Self-Proposed Rubrics: A novel reward mechanism for non-verifiable RL that decomposes judgment into binary criteria generated from pseudo-references, eliminating the need for human-annotated rubrics or reference answers.
2. Compute as Teacher (CaT) Framework: A general paradigm for turning inference compute into supervision via reference estimation and reward derivation. It demonstrates that synthesis (generative aggregation) can outperform selection-based baselines by reconciling distributed information across rollouts.
3. Empirical Validation: Comprehensive experiments across three model families (Gemma 3 4B, Qwen 3 4B, Llama 3.1 8B) on both non-verifiable (HealthBench) and verifiable (MATH-500) domains.

Results

• Performance Gains: CaT-trained models achieve up to +30% relative improvement on HealthBench and +33% on MATH-500 compared to the initial untrained policy.
• Efficiency: Trained models match or exceed the quality of inference-time aggregation (e.g., synthesis) while using 9× less test-time compute. The model learns to internalize the aggregation capability, requiring only a single forward pass at inference.
• Non-Verifiable Domains: Self-proposed rubrics perform comparably to expert physician-annotated rubrics on HealthBench and significantly outperform direct "Model-as-Judge" scoring.
• Synthesis vs. Selection: Synthesis-based reference estimation outperforms selection baselines (Majority Vote, Best-of-N) in non-verifiable domains. In cases where synthesis disagrees with the consensus, it is correct 70–86% of the time, often correcting errors present in all individual rollouts.
• RL vs. SFT: RL with rubric rewards consistently outperforms Supervised Fine-Tuning (SFT) on synthesized references, suggesting RL generalizes better from limited, synthesized data.

Significance and Claims

The paper claims that CaT provides a practical, "drop-in" solution for post-training LLMs in domains where reference answers are scarce, expensive, or non-existent. By leveraging inference compute to generate its own supervision, the framework:

• Bridges the gap between verifiable and non-verifiable RL, functioning as a unified approach.
• Reduces reliance on human annotation, offering a path forward for specialized domains where expert labeling is a bottleneck.
• Amortizes compute costs, shifting the heavy lifting of aggregation from inference time (repeatedly) to training time (once), resulting in more efficient deployment.

The authors note limitations, including the dependency on the base model's ability to generate diverse rollouts and the potential for performance plateaus as rollouts converge (entropy collapse). However, they position CaT as a significant step toward enabling reference-free learning in complex, qualitative domains.