Optimsyn: Influence-Guided Rubrics Optimization for Synthetic Data Generation

1. Problem Statement

Large Language Models (LLMs) rely heavily on high-quality Supervised Fine-Tuning (SFT) data to acquire problem-solving capabilities. However, in knowledge-intensive domains (e.g., medicine, law, humanities), high-quality real-world data is scarce due to:

• Cost and Expertise: Expert curation is expensive and time-consuming.
• Privacy: Strict constraints prevent the use of sensitive data.
• Consistency: Large-scale manual annotation is difficult to standardize.

Current Limitations of Synthetic Data:
Existing synthetic data pipelines typically use a "handcrafted rubric" approach:

1. Expert Dependency: Rubrics (rules/prompts) are designed by humans and are often domain-specific, failing to transfer across tasks or domains.
2. Brittle Optimization Loop: The process follows a heuristic loop: Write Rubric → Synthesize Data → Train Model → Inspect → Guess Tweaks. This lacks quantitative feedback, making it impossible to reliably attribute downstream performance gains to specific rubric choices.
3. Representation vs. Utility Gap: Recent work shows that synthetic data can be semantically close to real data in embedding space but fail to improve model performance because it steers optimization in suboptimal directions.

2. Methodology: OptimSyn Framework

The authors propose OptimSyn, a framework that replaces heuristic rubric engineering with a model-centric optimization loop driven by optimizer-aware influence estimation.

A. Core Insight: Influence over Embeddings

The paper demonstrates that embedding proximity does not correlate well with training utility. Instead, they utilize Influence Functions to quantify how a specific synthetic sample $(Q, A)$ affects the target model's validation loss.

• Metric: They use an Adam-compatible influence estimator ($IF_{Adam}$).
• Formula: The influence of a training sample $z$ on a validation sample $z'$ is approximated by the inner product of their gradients, adjusted for Adam optimizer moment statistics:
  $$Inf_{Adam}(z, z') = \sum_{i=1}^{T} \bar{\eta}_i \cos(\nabla_\theta \ell(z'; \theta_i), \Gamma(z, \theta_i))$$
  Where $\Gamma$ represents the Adam update direction.
• Validation: Experiments show a strong positive correlation between the aggregate influence score of a dataset and the downstream accuracy, validating it as a reliable proxy for data quality.

B. The Optimization Loop (RL Framework)

OptimSyn treats rubric generation as a learnable policy problem rather than a static design task.

1. Components:

   • Seed Data ($S$): Raw domain documents.
   • Prompter (Policy $\pi_\theta$): A model that generates a specific rubric $B$ for a given seed $S$.
   • Generator (Teacher): A strong LLM (e.g., Qwen-235B) that synthesizes a QA pair $(Q, A)$ conditioned on $(S, B)$.
   • Target Model: The model being fine-tuned (e.g., Qwen-8B).

2. Reward Function:
   The reward $R(\tau)$ for a trajectory $\tau = \{S, B, (Q, A)\}$ combines:

   • Validity Check: A lightweight verifier ensuring format, safety, and non-triviality ($Valid \in \{0, 1\}$).
   • Influence Score: The estimated contribution to the target model's objective.
   • Penalty: A penalty $\lambda$ for invalid generations.
     $$R(\tau) = Valid(Q, A) \cdot IF(Q, A) - \lambda(1 - Valid(Q, A))$$

3. Training Algorithm:

   • The Prompter is optimized using GRPO (Group Relative Policy Optimization).
   • For each seed, the policy generates $G$ candidate rubrics.
   • The advantage is computed via group normalization to reduce variance.
   • The policy updates to maximize the expected influence score, effectively learning rubrics that generate data most beneficial for the target model's specific optimization trajectory.

3. Key Contributions

1. Optimizer-Aware Influence Estimation: Adapted classical influence functions to modern LLM pipelines using Adam-compatible estimators. This provides a principled, gradient-based signal for assessing synthetic data utility, revealing a gap between semantic similarity and training impact.
2. From Heuristics to Supervision: Shifted rubric design from manual, expert-driven heuristics to a learnable policy optimized via Reinforcement Learning. The rubric generator learns to maximize downstream model improvement directly.
3. Broad Generalization & Portability: The framework requires no task-specific tuning. It successfully transfers across different domains (Humanities/Social Sciences, Medicine), target model families (Qwen, Llama), and generator models (Qwen, GPT, Gemini).

4. Experimental Results

The framework was evaluated on Humanities & Social Sciences (HSS) and Medical & Health domains.

• Performance Gains:
  • HSS Domain: OptimSyn upgraded the Qwen3-8B-Base model to outperform strong open SFT baselines (e.g., MAGACorpus, Bonito) and even rival the Qwen3-8B-Instruct model. Notable gains included a +27.2% relative improvement on the Humanity's Last Exam (HLE) benchmark.
  • Medical Domain: Outperformed specialized medical datasets (ChatDoctor, MedQA, Medical-o1) on benchmarks like HealthBench and HLE, narrowing the gap with instruction-tuned models.
• Generalization:
  • Model Agnostic: Consistent improvements were observed across Qwen variants (4B, 8B, 14B) and Llama3-8B.
  • Generator Robust: The method remained effective regardless of whether the generator was Qwen-235B, GPT-4.1, or Gemini-2.5-Pro.
• Data Efficiency:
  • Influence Correlation: Higher influence scores in synthetic subsets directly correlated with higher downstream accuracy, even with limited training budgets (2K–4K samples).
  • Quality over Quantity: The method achieved superior results with ~25k samples compared to baselines with significantly larger token counts, proving that "influence-aligned" data is more efficient than raw scale.

5. Significance and Impact

• Paradigm Shift: OptimSyn moves synthetic data generation from a "guess-and-check" heuristic process to a closed-loop, gradient-aligned optimization problem.
• Scalability in Data-Scarce Domains: It offers a viable solution for high-stakes fields (medicine, law) where data is private or expensive, allowing models to learn from synthetic data that is mathematically guaranteed to be useful for the specific target architecture.
• Automation: By automating rubric design via RL, it reduces the reliance on domain experts for data curation, making high-quality SFT data generation more accessible and portable across diverse tasks.

In conclusion, OptimSyn demonstrates that by aligning data synthesis with the actual optimization dynamics of the target model (via influence functions), one can generate synthetic data that is not just semantically plausible but functionally superior for training.