🤖 AI

Directing the Narrative: A Finetuning Method for Controlling Coherence and Style in Story Generation

This paper proposes a two-stage framework combining Group-Shared Attention (GSA) for intrinsic character consistency and Direct Preference Optimization (DPO) for style alignment, achieving state-of-the-art performance in story visualization by significantly improving character identity and style coherence on the ViStoryBench benchmark.

Original authors: Jianzhang Zhang, Yijing Tian, Jiwang Qu, Chuang Liu

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jianzhang Zhang, Yijing Tian, Jiwang Qu, Chuang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director filming a movie. You have a script (the story) and a lead actor (the character). Your goal is to shoot 20 different scenes: the hero walking in the rain, fighting a dragon, and having coffee in a cafe.

The Problem:
In the world of AI image generation, most "directors" (AI models) are terrible at keeping the lead actor consistent.

In Scene 1, the hero has a square jaw and a red scarf.
In Scene 2, suddenly the hero has a round face and a blue hat.
In Scene 3, the art style shifts from a cartoon to a photorealistic photo.

This is called "Identity Drift." The AI forgets who the character is because it treats every picture as a brand-new, isolated event, rather than part of a continuous story.

The Solution:
The paper "Directing the Narrative" introduces a new method to fix this. Think of it as a two-step training camp for the AI, designed to make it a master storyteller.

Step 1: The "Group Huddle" (Group-Shared Attention)

The Analogy: Imagine a classroom where every student is drawing a picture of the same character.

Old Way: Each student draws in total isolation. Student A draws a cat; Student B draws a dog. They never talk to each other.
New Way (GSA): The teacher (the AI) puts all the students in a circle. Before drawing, they all look at a "Reference Photo" of the character that is passed around the room.
How it works: The AI doesn't just look at the text prompt ("A cat on a roof"). It also looks at the other images in the same batch of generation. It uses a special mechanism called Group-Shared Attention (GSA) to "peek" at the clean, high-quality details of the reference character while it's drawing the new scene.
The Result: The AI learns, "Oh, I'm drawing the guy with the red scarf again. I need to make sure I copy that scarf exactly." It builds a structural memory of the character without needing to memorize a specific file for every single person.

Step 2: The "Tough Critic" (Direct Preference Optimization)

The Analogy: Now that the AI can keep the character looking the same, the drawings might still look a bit weird, stiff, or "off." Maybe the hands are too big, or the lighting is ugly.

Old Way: You try to fix this by adding math penalties (like telling the AI, "Don't draw too many fingers"). This often makes the AI confused and the pictures look worse.
New Way (DPO): You hire a human art critic. You show the critic two pictures:
1. A picture where the character looks great and consistent.
2. A picture where the character looks slightly distorted or the style is off.
  The critic says, "I prefer #1."
How it works: The AI learns from these choices. Instead of following a rigid math rule, it learns human taste. It understands that "good" means "consistent AND beautiful." This step polishes the rough draft from Step 1, fixing anatomical errors and making the story look like a professional movie.

The Final Result

By combining these two steps, the authors created a system that:

Remembers the character perfectly across different scenes (thanks to the "Group Huddle").
Looks beautiful and natural (thanks to the "Tough Critic").

Why it matters:
Before this, if you wanted to generate a comic book or a story with the same character in 50 different scenes, you had to do a lot of manual tweaking or accept that the character would look like a different person in every panel. This new method allows the AI to act like a true visual narrator, keeping the story coherent and the characters recognizable from start to finish.

In a nutshell: They taught the AI to look at its friends (other images) to remember who it is drawing, and then taught it to listen to human critics to make sure the drawing looks good. The result is a story that flows smoothly, just like a real movie.

1. Problem Statement

The paper addresses the critical challenge of Story Visualization: generating a sequence of images that align semantically with an evolving narrative while maintaining rigorous character identity and visual style consistency across frames.

Core Issues: Existing diffusion models (even advanced Diffusion Transformers like FLUX.1) suffer from identity drift (characters changing appearance, facial features, or clothing) and style inconsistency (disrupting the narrative flow) when generating multi-frame sequences.
Limitations of Current Methods:
- Optimization-based (e.g., LoRA, DreamBooth): Require computationally expensive per-subject fine-tuning, limiting scalability.
- Encoder-based (e.g., IP-Adapter): Rely on external vision encoders (like CLIP) to compress reference images. This creates an information bottleneck, leading to the loss of fine-grained identity details and semantic conflicts with text prompts.

2. Methodology

The authors propose a cohesive two-stage training framework built upon the pre-trained FLUX.1 backbone (a Rectified Flow Transformer). The goal is to inject consistency intrinsically rather than relying on external compression.

Stage 1: Structural Consistency via Group-Shared Attention (GSA)

This stage focuses on establishing a structural correspondence between frames without external encoders.

Group-Shared Attention (GSA): A novel mechanism that repurposes the self-attention layer. Instead of attending only to tokens within a single image, the model attends to high-resolution visual tokens from reference images within the same training batch.
- Mechanism: For a target image (index 0), the Key ( $K$ ) and Value ( $V$ ) matrices are extended to include visual tokens from $N-1$ reference images in the batch.
- Asymmetric Context: The target retains its full text-image context, while references contribute only visual tokens to prevent semantic conflict.
Training Strategy:
- Asymmetric Timestep Sampling: In a batch, the target image is assigned a noisy timestep ( $t > 0$ ), while all reference images are kept at $t=0$ (clean). This allows the model to query clean, high-fidelity features from references to reconstruct the noisy target.
- Objective: The model learns to predict the velocity field of the target conditioned on the clean reference latents, enabling lossless information flow.

Stage 2: Perceptual Refinement via Direct Preference Optimization (DPO)

While Stage 1 ensures structural consistency, outputs may still lack human aesthetic quality or contain subtle artifacts.

Approach: Instead of using conflicting auxiliary losses, the authors employ Direct Preference Optimization (DPO).
Data Construction: A curated dataset of "winners" (high-fidelity, consistent) and "losers" (distorted, inconsistent) is created based on human evaluation of character consistency, anatomical integrity, and visual aesthetics.
Training: An auxiliary adapter ( $\Phi_d$ ) is trained via DPO to align the model's output distribution with human preferences. Crucially, the Stage 1 consistency adapter ( $\Phi_c$ ) remains frozen to preserve the identity preservation capabilities while $\Phi_d$ refines the aesthetic quality.
Objective: The model maximizes the likelihood of preferred samples by minimizing the difference in flow matching errors relative to a reference policy.

3. Key Contributions

Group-Shared Attention (GSA): A novel mechanism that enables structured, lossless information flow across samples within a batch. It allows the model to intrinsically learn robust identity preservation from high-resolution internal features, eliminating the need for compressed external encoders.
Two-Stage Training Framework: A harmonized strategy combining structural pre-training (GSA) with preference-based refinement (DPO). This effectively balances the trade-off between rigid identity adherence and high-fidelity semantic generation.
State-of-the-Art Performance: The method achieves new benchmarks on the ViStoryBench dataset, significantly outperforming strong baselines in both character identity and style consistency.

4. Experimental Results

The method was evaluated on ViStoryBench (80 diverse story sequences) and compared against baselines like StoryGen, StoryDiffusion, StoryAdapter, and UNO.

Quantitative Improvements:
- Character Identity (CIDS): +10.0 improvement over the best baseline (UNO) in Cross-Sequence consistency (55.7 vs. 45.7).
- Style Consistency (CSD): +18.7 improvement over UNO (57.8 vs. 39.1).
- Prompt Adherence: Maintained high semantic alignment (OCCM score of 89.6), comparable to the best baseline.
Qualitative Analysis:
- Style Fidelity: Successfully suppressed the "realistic bias" of pre-trained models to adhere to stylized domains (e.g., flat 2D illustration, Western animation) without drifting into photorealism.
- Dynamic Scenes: Maintained identity persistence (e.g., specific beard shapes, coat styles) even during complex environmental shifts and multi-character interactions.
User Study: In a blind study with 30 participants, the proposed method scored significantly higher than all baselines in Character Consistency (4.60 vs. 3.48 for UNO) and Subjective Aesthetics (4.61).
Ablation Study:
- Removing GSA caused a catastrophic drop in identity preservation (CIDS dropped from 55.7 to 34.4).
- Removing DPO resulted in lower fidelity and anatomical artifacts, confirming the necessity of the refinement stage.

5. Significance

This paper presents a paradigm shift in story generation by moving away from external encoder dependencies toward intrinsic feature interaction.

Scalability: By avoiding per-subject optimization and external encoders, the method is more scalable for open-domain story generation.
Quality: It resolves the long-standing trade-off between identity consistency and visual fidelity, proving that preference optimization (DPO) can effectively refine structural consistency without compromising it.
Future Impact: The framework offers a modular, decoupled pipeline for consistent multi-sample generation, paving the way for controllable and coherent generative storytelling in complex narrative arcs.

Limitations: The computational cost of GSA scales with the number of reference images, which may hinder real-time applications for very long sequences. Additionally, maintaining consistency for highly abstract or non-anthropomorphic characters remains challenging. Future work aims to optimize attention complexity and extend the framework to video generation.