S-GRPO: Unified Post-Training for Large Vision-Language Models

1. Problem Statement

The paper addresses a critical bottleneck in adapting Large Vision-Language Models (LVLMs) to specialized domains (e.g., medical diagnostics, geometric reasoning). Current post-training paradigms suffer from inherent inefficiencies when used in isolation:

• Supervised Fine-Tuning (SFT): Forces the model to follow a single, deterministic expert trajectory. While efficient for knowledge injection, it causes catastrophic forgetting of the model's general multimodal capabilities due to severe distributional shifts (the "Alignment Tax").
• Reinforcement Learning (RL) / GRPO: Explores multiple trajectories to optimize against a reward function. However, in specialized visual tasks with strict binary verifiers, unaligned base models face a "cold-start" problem. They fail to spontaneously generate any valid trajectories, resulting in a group where all samples receive zero reward. This leads to vanishing gradients (advantage $\approx 0$) and optimization collapse, stalling learning entirely.

The core challenge is to unify the definitive guidance of SFT with the exploratory nature of RL in a single stage, preventing both catastrophic forgetting and optimization collapse.

2. Methodology: S-GRPO

The authors propose Supervised Group Relative Policy Optimization (S-GRPO), a unified framework that integrates imitation learning directly into the preference optimization process.

Core Mechanism: Conditional Ground-Truth Trajectory Injection (CGI)

The innovation lies in a dynamic fallback mechanism that modifies the sampling group during online training:

1. Standard Exploration: The policy samples a group of $G$ trajectories.
2. Failure Detection: A strict binary verifier evaluates the group. If all generated trajectories fail (i.e., receive a reward of 0), the system detects a "complete exploratory failure."
3. Conditional Injection: Upon detecting failure, the algorithm dynamically injects the verified ground-truth trajectory ($y_{gt}$) into the candidate pool, replacing one of the failed samples.
4. Reward Assignment: The injected ground-truth trajectory is assigned a deterministic maximal reward ($R_{max} = 1.0$), while the failed samples receive 0.
5. Advantage Calculation: This creates a mixed group with a non-zero standard deviation in rewards. The injected trajectory acts as a high-advantage anchor ($\hat{A}_{gt} \gg 0$), while the failed samples receive negative advantages. This guarantees a positive learning signal, preventing gradient vanishing.

Unified Objective

The S-GRPO objective function optimizes over this mixed group. As the model improves and begins generating valid trajectories autonomously, the CGI mechanism naturally phases out (since the failure condition is no longer met), allowing the training to seamlessly transition from supervised bootstrapping to pure self-exploratory RL without external curriculum design.

3. Key Contributions

1. Unified Post-Training Framework: S-GRPO eliminates the need for disjointed "SFT-then-RL" pipelines. It mathematically unifies the supervised learning objective (as a high-advantage component) with multi-trajectory exploration in a single on-policy stage.
2. Conditional Ground-Truth Injection (CGI): A novel strategy that solves the cold-start problem in sparse-reward visual tasks. It guarantees positive learning signals only when necessary, balancing the exploitation of expert trajectories with autonomous exploration.
3. Theoretical & Empirical Validation: The paper provides theoretical analysis showing how injection prevents gradient collapse and demonstrates empirical superiority over existing methods in both domain adaptation and general capability retention.

4. Experimental Results

The authors evaluated S-GRPO on three diverse datasets: COCO Caption (general), Geo170K (geometric reasoning), and OpenI (medical X-ray diagnostics), using Qwen2.5-VL-7B as the base model.

• Performance vs. SFT: S-GRPO achieved domain-specific performance comparable to or exceeding Full Fine-Tuning (FFT) (e.g., CIDEr scores of 1.0106 vs. 1.0172 on COCO). Crucially, unlike FFT, S-GRPO preserved general capabilities, avoiding the catastrophic forgetting seen in SFT (e.g., maintaining high IFEval-I scores).
• Performance vs. Pure RL: Standard GRPO failed completely on the base model due to the cold-start problem (yielding near-zero scores). S-GRPO successfully trained the model from this state.
• Performance vs. Two-Stage Pipelines: S-GRPO outperformed the computationally expensive "SFT $\to$ GRPO" pipeline in preserving general instruction-following capabilities while matching domain performance.
• Ablation Studies:
  • Unconditional Injection: Injecting ground truth in every step improved domain scores but degraded general capabilities due to excessive teacher-forcing bias.
  • Conditional Injection (CGI): Achieved the optimal Pareto balance, bootstrapping learning during cold-starts while allowing autonomous exploration once the model improved.
  • Threshold ($\delta$): A threshold of $\delta=0.9$ was found to be optimal, balancing strictness for domain learning with the need for autonomous exploration.

5. Significance

S-GRPO represents a significant advancement in LVLM alignment by resolving the "cold-start" dilemma in reinforcement learning for vision tasks.

• Efficiency: It removes the need for complex, multi-stage training pipelines or external teacher models, offering a computationally elegant single-stage solution.
• Stability: By guaranteeing positive advantage signals during exploration failures, it ensures stable convergence where standard RL fails.
• Capability Preservation: It demonstrates that domain specialization does not require sacrificing general-purpose intelligence, effectively mitigating the "Alignment Tax" associated with traditional fine-tuning.

In summary, S-GRPO provides a robust pathway for scaling specialized multimodal models, enabling them to learn rigorous domain constraints without losing their foundational multimodal understanding.