References Improve LLM Alignment in Non-Verifiable Domains

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has proven highly effective for reasoning tasks where ground-truth answers exist (e.g., math, coding). However, it cannot be directly applied to non-verifiable domains like LLM alignment (instruction following, helpfulness, safety) because reliable, rule-based verifiers do not exist for these open-ended tasks.

Currently, alignment relies on Reinforcement Learning from Human Feedback (RLHF) or AI Feedback (RLAIF), which typically use reference-free LLM-as-a-Judge evaluators. These evaluators often suffer from biases (positional, verbosity) and lack a "ground truth" anchor, leading to inconsistent supervision signals. The core question addressed by this work is: Can reference-guided LLM-evaluators act as "soft verifiers" to enable effective RL/alignment in non-verifiable domains without external human supervision?

2. Methodology

The authors propose a two-stage framework: Reference-Guided Evaluation and Reference-Guided Self-Improvement.

A. Reference-Guided LLM-Judges (Evaluation)

The authors argue that simply providing a reference output to an LLM judge is insufficient; explicit prompting strategies are required to leverage the reference effectively. They introduce two novel prompting protocols:

1. RefEval: Instructs the LLM judge to use the reference output as a benchmark for "successful instruction-following." The judge must verify factual correctness, adherence to constraints, and overall quality by comparing candidates against the reference.
2. RefMatch: Instructs the LLM judge to act primarily as a semantic and stylistic matcher, determining which candidate output demonstrates closer similarity to the reference output based on specific matching rules.

Experimental Setup for Evaluation:

• Judges: 11 diverse open-source LLMs (ranging from 7B to 72B parameters, e.g., Llama-3, Qwen-2.5, Gemma-2).
• References: Generated by a strong frontier model (GPT-4o) acting as an oracle.
• Datasets: 5 benchmarks including LLMBar (Natural/Adversarial), MTBench, Instrusum, and HREF.

B. Reference-Guided Self-Improvement (Training)

The authors apply the improved judges to a self-improvement pipeline for LLM alignment, avoiding the need for external reward models or human preference data.

1. Stage 1 (Distillation): Perform Supervised Fine-Tuning (SFT) on high-quality reference outputs (generated by DeepSeek-V3).
2. Stage 2 (Preference Optimization): Use Direct Preference Optimization (DPO) to further fine-tune the model.
   • Data Generation: Sample multiple outputs per instruction from the model.
   • Annotation: Use the Reference-Guided LLM-Judge (RefEval) to annotate preferences (selecting the best and worst outputs) to construct the DPO training pairs.
   • Goal: The model learns to align with the high-quality references using its own improved judgment as the supervisor.

3. Key Contributions

1. Novel Prompting Protocols: Demonstrated that naive inclusion of references yields modest gains, but explicit strategies like RefEval and RefMatch significantly enhance LLM-judge accuracy and inter-judge agreement.
2. Scalable Self-Improvement: Showed that high-quality references can replace expensive human/AI preference data for training reward models. The proposed method enables a "semi-self-improvement" loop where the model uses references to guide its own alignment training.
3. Performance Parity with Reward Models: Proved that reference-guided self-improvement achieves performance comparable to training with a specialized, fine-tuned reward model (ArmoRM), without the cost of training such a model.

4. Key Results

Evaluation Performance (LLM-as-a-Judge)

• Accuracy Gains: RefEval achieved an average accuracy of 79.1% across 11 judges and 5 datasets, significantly outperforming reference-free baselines (e.g., LLMBar-Base at 72.3%) and other reference-based methods (HREF-Ref at 74.8%).
• Impact on Smaller Models: Smaller models benefited most. For example, Llama-3-8B saw an absolute improvement of ~17.4% over the baseline when using RefEval.
• Inter-Judge Agreement: References increased pairwise agreement between different judges from 76.6% to 81.4%, indicating references provide a shared grounding that reduces decision variance.

Alignment Training Performance (Self-Improvement)

The authors fine-tuned Llama-3-8B-Instruct and Qwen2.5-7B using the proposed pipeline.

• vs. SFT Distillation: Reference-guided self-improvement (RefEval) significantly outperformed simple SFT on references.
  • Llama-3-8B: +19.2 points on AlpacaEval, +16.5 on Arena-Hard over SFT.
  • Qwen2.5-7B: +21.2 points on AlpacaEval, +17.6 on Arena-Hard over SFT.
• vs. Reference-Free Self-Improvement: RefEval consistently outperformed self-improvement using reference-free judges (RefFree).
  • Llama-3-8B: +5.6 points on AlpacaEval, +4.9 on Arena-Hard.
• vs. Fine-tuned Reward Models: The method achieved performance comparable to ArmoRM (a strong, pre-trained reward model).
  • Llama-3-8B Final Scores: 73.1 (AlpacaEval) and 58.7 (Arena-Hard).
  • Qwen2.5-7B Final Scores: 70.0 (AlpacaEval) and 74.1 (Arena-Hard).

Robustness

• Reference Quality: The method remains effective even with weaker reference generators (e.g., GPT-4o-mini), though performance is highest with frontier models (DeepSeek-V3).
• Task Categories: Improvements were most pronounced in Coding & Math and Reasoning tasks, with more variable gains in open-ended creative tasks.

5. Significance

This work bridges the gap between RLVR (which requires verifiable answers) and RLHF/RLAIF (which relies on subjective feedback) in non-verifiable domains.

• Cost Efficiency: It eliminates the need for expensive human preference annotation or the training of separate discriminative reward models.
• Practicality: It demonstrates that high-quality reference outputs (which are often easier to obtain than preference pairs) can serve as a powerful supervisory signal for post-training.
• Generalizability: The approach is model-agnostic and works across various model sizes and families, offering a scalable path for improving LLM alignment in complex, open-ended domains.

In conclusion, the paper establishes that reference-guided LLM evaluators are a viable and highly effective mechanism for enabling self-improvement in LLM alignment, achieving state-of-the-art results without external reward model training.