Technical Summary: Stable-Layers

Problem Statement

Image layer decomposition—the task of separating a single RGB image into a small set of editable RGBA layers whose composition reconstructs the original—is a fundamental primitive for professional editing. While the task is conceptually simple, it is inherently difficult to supervise. A single image admits many plausible decompositions, and quality is determined by downstream usability (semantic separation, clean alpha mattes, minimal redundancy, and faithful handling of occluded content) rather than similarity to a single ground-truth target.

Existing methods typically rely on synthetic layered datasets or paired supervision (e.g., Photoshop PSD files). However, such approaches impose a limitation: when multiple decompositions are equally valid, regression toward a single target penalizes alternative, equally plausible solutions. Furthermore, natural images lack paired ground-truth layer annotations, creating a supervision gap for training robust decomposition models on real-world data.

Methodology

The authors propose Stable-Layers, a reinforcement learning (RL) framework that fine-tunes a pretrained layer decomposition model using only unlabeled images and feedback from a Vision-Language Model (VLM), eliminating the need for paired supervision or synthetic targets.

Core Framework

The method builds upon Flow-GRPO (Group Relative Policy Optimization for Flow Matching) and the Qwen-Image-Layered base model. The training loop operates in three phases:

1. Generation: For a given input image, the policy samples $G$ candidate decompositions via SDE-augmented flow matching.
2. Scoring: A VLM judge evaluates the candidates using a specialized two-phase reward protocol.
3. Optimization: The policy is updated using GRPO, optimizing for group-relative advantages derived from the VLM scores.

Key Technical Innovations

1. Two-Phase VLM Reward Protocol
The primary challenge in applying VLM-as-judge feedback is that VLMs tend to compress their judgments into a narrow score band when evaluating samples in isolation, resulting in low within-group variance that weakens the GRPO learning signal. To address this, Stable-Layers employs a two-stage evaluation pipeline:

• Phase 1 (Structured Individual Scoring): Each candidate is scored independently on a 0–5 scale across five edit-centric criteria: semantic separation, alpha cleanliness, background inpainting, feature distribution, and content validity. Anchored rubrics ensure consistent evaluation of categorical failures.
• Phase 2 (Relative Grid Calibration): To restore variance, all $G$ candidates are tiled into a labeled comparison grid. The VLM re-scores the candidates relative to one another within this grid. This step spreads the compressed scores, providing the fine-grained relative discrimination necessary for effective advantage normalization in GRPO.

2. Adaptation for Packed Latents (RatioNorm Reformulation)
Standard GRPO-Guard stabilizes training by normalizing the importance ratio of log-probabilities. However, the Qwen-Image-Layered model packs $N$ RGBA layers into a single high-dimensional latent sequence, inflating the effective dimensionality $D$ by approximately 5×. This causes the standard spatial-mean log-ratio to collapse toward zero, suppressing the learning signal.
The authors introduce a sum-and-rescale modification: instead of taking a spatial mean, they sum log-probabilities over spatial dimensions and normalize by $\sqrt{D}$ before applying per-step centering. This restores $O(1)$ ratio magnitudes while retaining the variance-stabilization properties of GRPO-Guard.

3. Data Strategy
The model is trained entirely on Fine-T2I, an aesthetically filtered subset of unlabeled images. No layer annotations, paired examples, or synthetic decomposition targets are used. The number of output layers is sampled uniformly (typically 2–5) during training to expose the model to variable decomposition complexity.

Key Contributions

1. A Two-Phase VLM Reward Protocol: A method to alleviate score compression in within-group reinforcement learning by combining structured absolute scoring with relative grid calibration, enabling richer credit assignment over layer stacks.
2. RatioNorm Reformulation: A modification tailored to packed latent representations in flow-matching RL, ensuring stable optimization for models generating multiple layers simultaneously.
3. Improved Decomposition Quality: A demonstration that RL fine-tuning with VLM judges can significantly improve layer separation, reduce artifacts, and enhance background inpainting beyond what is achievable with supervised training alone.

Results

Evaluation on the Crello dataset and held-out LAION-Aesthetics images demonstrates substantial improvements over the base Qwen-Image-Layered model:

• Quantitative Metrics: Stable-Layers achieves lower per-layer RGB L1 reconstruction error across all tested layer counts (2, 3, and 4). It significantly reduces the occurrence of "bad" layers (blank or artifact-heavy) from ~1.65 to ~0.4 per decomposition.
• Qualitative Improvements:
  • Background Inpainting: The base model often produces degenerate backgrounds (e.g., solid black or flat fills). Stable-Layers generates plausible scene completions where foreground objects are removed.
  • Semantic Separation: The base model tends to duplicate the composite image across foreground layers. The fine-tuned model isolates distinct semantic elements (e.g., separating a person from a bridge railing) onto separate layers with cleaner alpha masks.
  • Feature Distribution: Content is more evenly distributed across layers rather than being concentrated in a single layer.
• Ablation Studies:
  • Grid Calibration: Removing Phase 2 (grid calibration) results in lower Layer 0 quality (inpainting plausibility) and edge sharpness, confirming that relative calibration is essential for fine-grained improvements.
  • Text Conditioning: Using a detailed prompt mirroring the reward rubric actually degraded performance compared to a basic prompt, suggesting that explicit prompt conditioning may conflict with the VLM judge's evaluation axes.

Significance and Claims

The paper positions Stable-Layers as a general recipe for training edit-oriented generators using judge feedback instead of ground-truth targets. The authors claim that their approach successfully converts black-box VLM judgments into learning signals that drive fine-grained improvements in layer separation and content validity without requiring task-specific reward training or human annotation.

The work highlights that while VLMs are powerful judges, their raw scoring requires specific calibration (via grid comparison) to be effective in RL settings where within-group variance is critical. The proposed method offers a pathway to improve generative models for editing tasks (such as style transfer, inpainting, or scene rearrangement) where paired ground-truth data is unavailable or where multiple valid outputs exist.

Limitations: The authors note that the method relies on a proprietary VLM (Gemini-3-flash-preview), introducing API costs and potential score distribution drift across model versions. Evaluation relies on automated metrics and qualitative inspection rather than human studies. Additionally, training was capped at five layers per sample due to compute constraints, leaving the behavior on high-layer-count decompositions (supported by the base model up to 20) unevaluated.