Technical Summary: Training Skills Like Parameters via Self-Supervised Semantic Diffusion

Problem Statement

Large Language Models (LLMs) excel at general instruction following but often plateau in highly specialized, open-ended domains like creative screenwriting. Traditional adaptation methods face significant hurdles:

• Parameter-level adaptation: Fine-tuning closed-source frontier models is impossible due to weight inaccessibility, while full fine-tuning is computationally prohibitive. Furthermore, learned capabilities are tied to specific checkpoints and are opaque to human auditing.
• Externalized skill injection: While injecting external "skill" documents is a promising alternative, current approaches rely heavily on costly human expert annotations to craft these skills. Domain experts are often reluctant to formalize implicit knowledge, creating a bottleneck for automated skill acquisition.
• Feedback limitations: Existing agentic continual learning methods often depend on unreliable "LLM-as-a-judge" feedback or expensive human ratings, which struggle to provide robust signals for open-ended generation tasks.

Methodology

The authors propose a self-supervised, self-evolving agent framework inspired by the corruption-and-reconstruction paradigm of diffusion models. Instead of updating model weights, the system optimizes an external library of textual skills (memory).

Core Loop

The framework operates on a cycle of corruption, reconstruction, loss calculation, and credit assignment:

1. Corruption (Self-Supervised Pair Construction):

   • High-quality, production-grade human screenplays are treated as the "clean" state.
   • A corruption agent compresses these scripts into condensed story outlines, stripping away dialogue, pacing, and physical actions. This creates a "noised" input where the story skeleton remains but craft-level details are destroyed.
   • The (outline, original script) pair serves as a free training example without external labels.

2. Forward Pass (Script Generation):

   • Given an outline, a script agent reconstructs a full Hollywood-format screenplay.
   • Before writing, the agent scans a library of rule cards (skills) and reads those deemed relevant. A "read trace" is logged to identify which rules influenced the generation.

3. Semantic Loss (Contrastive Signal):

   • A loss agent compares the reconstructed script against the original human artifact.
   • Instead of token-level matching (which is ineffective for creative writing), the agent generates a dense textual loss report highlighting structural and stylistic discrepancies (e.g., how a confrontation escalates, where an episode ends).
   • This provides a robust self-supervised signal without requiring external human ratings.

4. Backward Pass (Credit Assignment & Memory Update):

   • A backward agent acts as the optimizer, reading the loss report and the read trace.
   • It performs fine-grained edits to the memory library: strengthening effective rules, narrowing the scope of misleading ones, creating new rules for recurring errors, and merging redundant items.
   • Hierarchical Reduction: To prevent signal dilution, loss reports from multiple novels are first reduced into cross-novel commonality summaries before being processed.
   • Micro-steps: Global updates are decomposed into short, focused micro-steps to avoid vanishing gradients and ensure traceability.

Memory Architecture

• Mixture-of-Experts (MoE) Structure: The memory library is organized into three expert folders: General Principles, Concrete Craft Techniques, and Genre Conventions. Rule cards are dynamically routed to these folders.
• Regularization (The Linter): A strict linter enforces hard length limits (e.g., 1,200 tokens per card body) to prevent rules from overfitting to specific plot details or growing into "incident logs."
• Data-Parallel Training: Forward generation and loss computation run in parallel across novels, while backward updates are serialized into a queue of micro-steps.

Key Contributions

1. Unsupervised Skill Acquisition: The framework enables agents to autonomously extract and internalize generalizable skills from existing human artifacts without any human annotation or explicit reward signals.
2. Semantic Diffusion Paradigm: It adapts the corruption-reconstruction logic of diffusion models to the textual domain, treating skill documents as trainable parameters that evolve via a semantic loss.
3. Textual Credit Assignment: It introduces a mechanism to route textual feedback into specific memory nodes using a MoE-style architecture, effectively shifting gradient updates from the parameter space to external skill documents.
4. Scalable Self-Evolution: By utilizing micro-steps and hierarchical reduction, the system scales to train on large corpora (20 novels, ~1,100 episodes) while maintaining a compact, stable memory pool (18 rule cards).

Experimental Results

The framework was evaluated on the task of short drama screenwriting using 20 professional novels for training and 6 held-out novels for Out-of-Distribution (OOD) evaluation.

• Performance Gains: The memory-equipped agent outperformed the baseline (empty memory) on 7 out of 9 metrics.
  • Unfilmable Action: Reduced from 1.1 lines/episode (baseline) to 0.3, matching the human baseline of 0.3.
  • Hard-Cut Endings: Hit rate increased from 40% (baseline) to 59%, approaching the human rate of 61%.
  • Cold Opens: Increased from 64% to 84%, surpassing the human reference of 72%.
  • Prop Activation: Improved from 59% to 78%, exceeding the human rate of 70%.
• Training Dynamics: The agent showed monotonic improvement in dialogue pacing and closed-loop correction in voice-over usage, demonstrating the system's ability to self-correct over epochs.
• Cross-Model Transfer: A memory pool trained exclusively on Claude Sonnet 4.6 was successfully transferred to Claude Opus 4.8. The pool guided both models to nearly identical performance levels on craft disciplines, indicating the skills are model-agnostic.
• External Pipeline Validation: When injected into a third-party multi-agent screenwriting pipeline unseen during training, the memory successfully corrected specific failures (e.g., reducing excessive voice-overs, enforcing hard cuts), proving robustness outside the native training loop.
• Regularization Impact: Experiments showed that only the token-capped memory architecture (strict length limits) successfully improved both rule-based and judge-based metrics simultaneously. Uncapped or line-limited pools either over-suppressed content or collapsed on rule metrics.

Significance and Claims

The paper claims to offer a scalable pathway for agents to teach themselves the production of complex, high-quality human artifacts without external supervision.

• Auditability: Unlike gradient-based learning, the learned "skills" are explicit, human-readable text documents that can be inspected and audited.
• Portability: The skills are model-agnostic and can be transferred across different model generations and architectures.
• Cost-Efficiency: By leveraging existing high-quality human artifacts as self-supervised signals, the method bypasses the need for expensive human expert annotation or reinforcement learning loops.

The authors acknowledge limitations, specifically the reliance on the base model's ability to follow instructions (if the model ignores a rule, the credit assignment chain breaks) and the retrieval mechanism (if a rule is never read, it cannot fire). Future work focuses on optimizing memory architecture and improving the credit assignment mechanism. The authors intend to open-source the training framework and representative memory examples.