Technical Summary: SAGE (Scalable Automated Robustness Augmentation for LLM Knowledge Evaluation)

Problem Definition

Large Language Models (LLMs) currently demonstrate high accuracy on standard knowledge evaluation benchmarks (e.g., MMLU, CommonsenseQA). However, recent research indicates that these models remain "brittle" when faced with question variants that test the same underlying knowledge but alter the formulation or reasoning process (e.g., negation, causal inference, or scenario refinement).

Existing efforts to augment benchmarks with such variants, such as HellaSwag-Pro, rely on a "generate-then-verify" pipeline using strong, proprietary LLMs. This approach faces two critical bottlenecks:

1. Low Yield: Strong models frequently generate invalid or low-quality variants, leading to high token costs (only ~46% of candidates survived quality control in HellaSwag-Pro).
2. Verification Reliability: Prompted LLMs are unreliable at judging variant validity, necessitating extensive human annotation to filter candidates.

These bottlenecks limit the scalability and cost-effectiveness of creating large-scale robustness benchmarks.

Methodology: The SAGE Framework

The authors propose SAGE (Scalable Automated Generation of Robustness BEnchmarks), a framework designed to automate robustness augmentation using fine-tuned smaller models rather than costly prompting of strong models. SAGE consists of two primary components and a three-stage construction pipeline.

1. Core Components

• VariantQual (Verifier): A rubric-based verifier trained to evaluate the quality of generated variants.
  • Rubric Design: The authors decompose variant validity into three dimensions:
    1. Type Compliance (TC): Does the variant preserve the original knowledge while satisfying the specific variant type constraints?
    2. Label Correctness (LC): Is the annotated label correct given the new context and choices?
    3. Answer Uniqueness (AU): Is there exactly one correct answer, with no duplicates or irrelevant distractors?
  • Training Strategy: The verifier is fine-tuned on human-labeled seed data from HellaSwag-Pro. The paper compares Explicit Rubric Aggregation (ERA), which predicts each dimension separately, against Implicit Rubric Aggregation (IRA), which predicts the final validity label jointly. Empirical results favored IRA for better overall accuracy.
• VariantGen (Generator): A model responsible for generating question variants based on an original question and a target variant type.
  • Initialization: Initialized via Supervised Fine-Tuning (SFT) on human-annotated seed examples.
  • Optimization: Further optimized using Group Relative Policy Optimization (GRPO). In this stage, VariantQual serves as the reward model, providing validity scores to guide the generator toward high-quality outputs without requiring human feedback during the RL phase.

2. Benchmark Construction Pipeline

The framework operates in three stages:

1. SFT: Both VariantGen and VariantQual are fine-tuned on human-annotated data (using Qwen3-0.6B as the backbone).
2. RL Optimization: VariantGen is optimized using GRPO, where VariantQual provides reward signals based on the rubric dimensions.
3. Large-Scale Generation & Filtering: The system generates candidates for all seven predefined variant types (mapped to Bloom's cognitive taxonomy). VariantQual filters these candidates, retaining only those that pass all three rubric dimensions. This process repeats until the target number of accepted examples is reached.

Key Contributions

1. Task Formalization: The paper formalizes the task of scalable automated robustness augmentation, defined as generating valid question variants under predefined types.
2. SAGE Framework: A novel framework that replaces expensive strong-model prompting with a pipeline of fine-tuned small-model generators and evaluators. This approach improves both cost-efficiency and construction accuracy.
3. Empirical Validation: The application of SAGE to HellaSwag to construct a 16,800-sample robustness-augmented benchmark. The framework demonstrates cross-benchmark generalization to MMLU without additional fine-tuning.

Experimental Results

The authors evaluated SAGE on HellaSwag and MMLU across 12 LLMs from four families (Gemma, Llama, Qwen, GPT).

• Generator Performance: The SAGE-generated variants achieved a pass rate of 91.80% (SFT + GRPO), outperforming a prompt-based Qwen2.5-Max baseline (which achieved ~90% on average but varied by type). SAGE exceeded the baseline on all seven variant types, with particularly strong performance on complex semantic transformations like Scenario Refinement.
• Verifier Performance: The VariantQual model (Qwen3-0.6B) achieved 90.21% accuracy in validating variants. The study found that larger models (1.7B, 4B) did not significantly outperform the 0.6B model, suggesting the task is sufficiently constrained for smaller architectures.
• Benchmark Quality:
  • Correlation: The SAGE-generated benchmark preserved the relative ordering of model robustness established by human-annotated HellaSwag-Pro references. Spearman correlation coefficients were 0.857 for Average Robust Accuracy (ARA) and 0.964 for Consistent Robust Accuracy (CRA).
  • Robustness Findings: Experiments confirmed that robustness lags behind capability. While original accuracy (OA) ranged from 53% to 74%, robust accuracy (ARA) was compressed to 31%–37%. Scaling model size increased Robustness Loss Accuracy (RLA), indicating that capability gains do not proportionally translate to robustness gains.
• Cost Efficiency: The total monetary cost for the reported experiments was approximately $284 USD, compared to an estimated $7,064 USD for the human annotation process in HellaSwag-Pro, representing a ~25x cost reduction.

Significance and Claims

The paper claims that SAGE facilitates a shift from static, manually constructed benchmarks toward scalable, continuously updated robustness diagnostics. By automating the generation and verification pipeline using small, fine-tuned models, SAGE lowers the financial and logistical barriers to creating high-quality robustness benchmarks. This enables resource-limited research groups to participate in LLM evaluation and provides a mechanism to systematically probe model weaknesses under controlled perturbations.

The authors maintain a modest stance, noting that while SAGE significantly reduces reliance on human annotation, it does not fully replace it for the most semantically complex variants (e.g., Scenario Refinement), where strict compliance rates are lower. Furthermore, the framework is currently limited to multiple-choice knowledge evaluation and may require adaptation for free-form generation or multimodal tasks.