Technical Summary: AttriStory: Fine-grained Attribute Realization for Visual Storytelling with Diffusion Models

1. Problem Definition

While recent advances in visual storytelling using diffusion models have achieved significant success in maintaining character consistency across multiple narrative scenes (e.g., ensuring a character's identity and appearance persist), a critical gap remains regarding fine-grained attribute realization. Existing methods often fail to faithfully render specific, detailed visual attributes specified in the narrative text, such as clothing colors, fabric textures, and accessories.

The authors define this as the attribute realization problem: ensuring that explicitly specified visual attributes (e.g., "red shirt," "blue sneakers," "brown corgi") are correctly bound to their corresponding objects in the generated images. This challenge is fundamentally orthogonal to character consistency; a model may successfully preserve a character's identity across scenes but fail to render the specific attributes described in the prompt for each scene. Current benchmarks and evaluation metrics largely neglect these details, focusing instead on basic character identity and actions.

2. Methodology

2.1. The AttriStory Benchmark

To address the lack of systematic evaluation for attribute realization, the authors introduce AttriStory, a new benchmark designed specifically for this purpose.

• Dataset Construction: The benchmark consists of 200 multi-scene stories across 10 distinct artistic styles (including photo, cartoon, 3D animation, watercolor, oil painting, etc.). Each story contains 5 scenes.
• LLM-Driven Generation: Large Language Models (LLMs) are used to generate coherent narratives with structured, detailed specifications. For each scene, the LLM produces:
  • Character Descriptions: Detailed multi-attribute specifications.
  • Scene Narratives: Text enriched with fine-grained visual attributes.
  • Positive Attribute-Object Pairs ($P^+$): Pairs that must co-occur (e.g., ["pink", "dress"]).
  • Negative Attribute-Object Pairs ($P^-$): Pairs that must not co-occur to prevent spurious associations (e.g., ["pink", "lilies"]).
• Validation: The dataset undergoes manual review to ensure diversity in attributes, correctness of pairs, and consistency across scenes.

2.2. AttriLoss: Latent Optimization Module

To solve the attribute realization problem, the authors propose AttriLoss, a plug-and-play latent optimization module that operates during the early denoising steps of the diffusion process (specifically steps 1–25 out of 50). This timing is chosen because early steps are critical for establishing structural and semantic content, including colors and textures.

• Mechanism: The method leverages cross-attention maps within the diffusion model. These maps visualize the association between text tokens and spatial image regions. The authors observe that ambiguous overlaps in these maps (e.g., the token "pink" attending to "lilies" instead of "dress") lead to incorrect attribute binding.
• Objective Function: AttriLoss optimizes the spatial overlap (Intersection-over-Union, or IoU) between attention maps of attribute-object token pairs:
  • Maximize IoU for positive pairs ($P^+$) to encourage correct co-occurrence.
  • Minimize IoU for negative pairs ($P^-$) to suppress spurious associations.
  • The loss function is formulated as:
    $$L_{Attri} = -\sum_{(i,j) \in P^+} \text{IoU}(A_i, A_j) + \sum_{(i,j) \in P^-} \text{IoU}(A_i, A_j)$$
    where $A_i$ and $A_j$ are the spatial cross-attention maps for tokens $i$ and $j$.
• Optimization: The latent code $z_t$ is updated via gradient descent using this loss: $z'_t \leftarrow z_t - \nabla_{z_t} L_{Attri}$.
• Integration: This approach is orthogonal to existing consistency mechanisms (like self-attention extensions). It integrates seamlessly with current training-free pipelines (e.g., ConsiStory, StoryDiffusion) without requiring architectural modifications or retraining.

3. Key Contributions

1. AttriStory Benchmark: A comprehensive dataset of 200 multi-scene stories with explicit positive and negative attribute-object annotations across 10 artistic styles, enabling systematic evaluation of fine-grained attribute realization.
2. AttriLoss: A targeted IoU loss applied to cross-attention maps during early denoising steps. It guides models to correctly localize attributes by maximizing alignment for desired pairs and suppressing undesired ones.
3. Plug-and-Play Integration: A method that complements existing character consistency mechanisms, allowing immediate adoption in current story generation pipelines without architectural changes.
4. Empirical Validation: Demonstration that attribute realization is a distinct, complementary dimension to character consistency, advancing the field toward fine-grained visual narrative generation.

4. Experimental Results

The authors evaluated AttriLoss on three baselines: Vanilla SDXL, StoryDiffusion, and ConsiStory, using the AttriStory benchmark.

• Quantitative Metrics:
  • VQAScore: Showed significant improvements across all baselines (e.g., ConsiStory + AttriLoss improved from 0.8136 to 0.8490), indicating better adherence to fine-grained attribute prompts.
  • CLIP-T (Image-Text Similarity): Showed modest or consistent gains, suggesting that AttriLoss refines attribute specificity without disrupting global semantic alignment.
  • CLIP-I (Cross-Scene Consistency): Scores remained strong or improved, confirming that attribute grounding does not compromise character consistency.
  • DreamSim (Perceptual Quality): Improved across all methods, reflecting enhanced holistic narrative quality.
• Qualitative Analysis: Visual results demonstrated that while baselines maintained character consistency, they often failed to bind specific attributes (e.g., rendering "white lilies" as "pink roses" or failing to color an umbrella correctly). With AttriLoss, these fine-grained specifications were faithfully realized across diverse artistic styles while preserving character identity.

5. Significance and Claims

The paper positions attribute realization as a distinct and complementary dimension of visual storytelling, alongside character consistency. The authors claim that their work bridges the gap between identity preservation and attribute control, providing both a systematic benchmark and a practical optimization method.

The significance lies in addressing a critical need for professional creative workflows (e.g., animation, editorial illustration) where precise control over visual details like clothing and accessories is essential for high-quality storytelling. By introducing AttriLoss, the authors enable diffusion models to move beyond basic character consistency toward fine-grained attribute-controlled story generation, operating orthogonally to existing methods to enhance fidelity without requiring retraining.

Limitations: The authors acknowledge that while attribute realization improves, the method does not guarantee the realization of complex actions described in the text (e.g., a character reading a book or steering a raft), indicating scope for future improvement in action grounding.