Technical Summary: Self-Distillation Policy Optimization via Visual Feedback

1. Problem Statement

Code-generating Large Language Models (LLMs) are increasingly tasked with producing visual artifacts (charts, web pages, slides) by writing executable code. A fundamental disconnect exists between the model's optimization target (syntactically valid code) and the user's evaluation target (the rendered visual output). Because the rendering process is non-differentiable, the model must commit to code before observing the visual result. Consequently, even executable code frequently produces artifacts with salient visual defects, such as overlapping elements, clipped text, broken alignment, low contrast, and overflow.

Existing approaches to bridge this "code–vision gap" face significant limitations:

• Inference-time visual reflection: Iterative render–critique–revise loops require re-executing renderers and querying critics at inference time, leading to high deployment costs and latency.
• Visual-reward Reinforcement Learning (RL): Methods optimizing against image-level scalar rewards (e.g., via GRPO or DPO) suffer from sparse, trajectory-level supervision. They collapse per-element diagnostics into a single number before gradients can be shaped, losing spatial granularity.

2. Methodology: Visual-SDPO

The paper proposes Visual-SDPO, a self-distillation policy-optimization framework that treats rendered visual feedback as "privileged context" for a weight-sharing teacher model, distilling this feedback into a coding student model that operates without the renderer at inference.

2.1 Visual-Feedback Self-Distillation

The framework employs a teacher-student architecture where both share weights ($\pi_\theta$):

• Student: Generates a code rollout $y$ conditioned only on the original prompt $x$.
• Teacher: Re-scores the student's tokens conditioned on the prompt $x$ plus privileged visual context $v$. The context $v$ is derived from the student's own rollout:
  • Image Channel: The raw rendered artifact (or execution error messages if rendering fails).
  • Rubric Channel: A structured binary defect table (e.g., alignment, overlap, color) extracted from the artifact.
• Objective: The student minimizes the Kullback-Leibler (KL) divergence between its token distribution and the teacher's distribution, which has access to the visual outcome. This transfers visual understanding into the student without requiring a renderer during inference.

2.2 Visual-Grounded Code Credit Weighting

Standard token-level distillation treats all output tokens equally, which can induce policy drift on non-visual scaffolding (e.g., imports, control flow). To address this, the authors introduce Visual-Grounded Code Credit Weighting, which spatially targets supervision:

1. Defect Detection: A detector identifies visual defects $D$ (regions and severity) on the rendered artifact.
2. Region-to-Code Mapping ($M$): A mapping mechanism traces each defect region back to the specific code statements responsible for the affected elements. This is achieved via:
   • Runtime Introspection: Instrumenting renderers (matplotlib, Playwright, python-pptx) to record source statements for each element.
   • VLM-based Mapping: Using the teacher VLM to identify responsible statements from code and annotated images.
3. Statement-Weighted Loss: A responsibility score $resp(s)$ is computed for each statement $s$ based on the overlap (IoU) between its generated regions and detected defects. This score is converted into a per-token weight $w_t$:
   $$w_t = 1 + (\alpha - 1) \cdot resp(s(t))$$
   where $\alpha \ge 1$ is an amplification factor. Tokens in statements causing defects receive higher weights, concentrating the distillation gradient on the specific code responsible for visual errors.

2.3 Final Training Objective

The framework combines the dense, localizable token-level distillation loss with a sequence-level Group Relative Policy Optimization (GRPO) reward:
$$L(\theta) = L_{VSDPO}(\theta) + \beta \cdot L_{GRPO}(\theta)$$

• Distillation Term ($L_{VSDPO}$): Provides dense, sample-efficient supervision weighted by visual defects.
• GRPO Term ($L_{GRPO}$): Uses a multiplicative reward $r(y) = r_{exec}(y) \cdot r_{vis}(y)$, where $r_{exec}$ ensures code executability and $r_{vis}$ is a graded score based on the rubric extractor. This anchors training across domains and provides a signal even when code fails to execute (via error message feedback).

3. Key Contributions

1. Visual-SDPO Framework: A self-distillation approach for code-generated visual artifacts where a weight-sharing teacher observes rendered artifacts as privileged context, distilling visual feedback into a student that requires no renderer at inference.
2. Visual-Grounded Code Credit Weighting: A mechanism that traces visual defects back to responsible code statements to weight token-level distillation, ensuring gradients focus on spatially relevant code rather than uniform token matching.
3. Unified Evaluation: A comprehensive instantiation and evaluation across three distinct domains (charts, web/UI, slides) using a single Qwen3-VL-8B-Instruct backbone, demonstrating the method's versatility.

4. Experimental Results

The method was evaluated on ChartMimic (charts), Design2Code (web/UI), and AeSlides (slides).

• Performance Gains: Visual-SDPO improved over the zero-shot base by more than 10 absolute points in primary metrics across all benchmarks.
  • ChartMimic: +10.7 points over zero-shot; +1.6 points over the strongest single-signal baseline (OPSD).
  • Design2Code: +10.5 points over zero-shot; +2.6 points over GRPO.
  • AeSlides: +11.2 points over zero-shot; +2.5 points over GRPO.
• Ablation Insights:
  • Complementarity: The combination of dense token-level distillation (with credit weighting) and sequence-level GRPO outperformed either signal alone. Token-level supervision was particularly effective for low-level structural details (e.g., matplotlib APIs, CSS properties), while GRPO anchored the overall trajectory.
  • Efficiency: Visual-SDPO achieved performance comparable to full GRPO training with significantly fewer rollout generations (approx. 29% of GRPO's budget) and less wall-clock time, demonstrating superior sample efficiency.
• Backbone: All experiments utilized a unified Qwen3-VL-8B-Instruct backbone.

5. Significance and Claims

The paper positions Visual-SDPO at the intersection of privileged self-distillation, visual-feedback training, and code-generated visual artifacts. Its primary significance lies in:

• Bridging the Gap: It effectively addresses the code-vision gap by using rendered feedback as a privileged signal without incurring inference-time costs.
• Granular Supervision: Unlike scalar reward methods, it provides localizable, spatially targeted feedback, identifying exactly which code statements cause specific visual defects.
• Stability and Efficiency: By combining dense distillation with trajectory-level rewards, it achieves stable training that is more sample-efficient than pure RL approaches.

The authors note that this approach avoids the need for external VLM critics or renderers during inference, making it suitable for deployment in resource-constrained environments while maintaining high visual fidelity in generated artifacts.