Caption Generation for Dongba Paintings via Prompt Learning and Semantic Fusion

This paper introduces PVGF-DPC, an encoder-decoder framework that leverages prompt learning and a visual semantic-generation fusion loss to overcome domain shifts and generate culturally accurate captions for Dongba paintings, supported by a newly constructed dataset of 9,408 annotated images.

The Gist

Imagine you have a magical, ancient book of pictures called Dongba paintings. These aren't just pretty drawings; they are the visual diary of the Naxi people from southwestern China. Every line and color tells a story about gods, ghosts, rituals, and myths.

However, there's a problem: Computers are terrible at reading these stories.

If you show a standard AI (trained on photos of cats, cars, and beaches) a Dongba painting, it gets confused. It might see a tiger and say, "A tiger in a zoo," when it's actually a sacred guardian beast of a temple. It's like trying to explain a complex Shakespeare play to someone who only knows how to read street signs. The AI misses the cultural "flavor" and makes things up (hallucinations).

This paper introduces a new AI system called PVGF-DPC to fix this. Here is how it works, using simple analogies:

1. The Problem: The "Tourist" vs. The "Local"

Think of standard AI models as tourists. They have a big map of the world (natural images), but when they visit a specific village (Dongba art), they don't speak the local dialect or understand the customs. They guess based on what they've seen elsewhere, which leads to wrong answers.

The authors realized they needed a local guide who knows the history, symbols, and language of the Naxi culture.

2. The Solution: The "Smart Guide" System

The new system, PVGF-DPC, acts like a team of experts working together:

• The Eyes (MobileNetV2 Encoder): This is the part that looks at the painting. Instead of just seeing "lines and colors," it's trained to spot specific cultural details, like a "deity sitting on a lotus" or a "white bat with a sacred mission."

• The Local Guide (Content Prompt Module): This is the magic trick. Before the AI tries to write a sentence, this module looks at the picture and asks: "What kind of story is this?"

  • Is it about a God?
  • Is it about a Ghost?
  • Is it about Music?
  • Is it about Fishing?

  Once it figures out the category, it whispers a hint (a "prompt") to the writer. It's like a teacher giving a student a topic before an essay: "Okay, now write a story about a deity." This stops the AI from guessing wildly and keeps it on the right cultural track.

• The Writer (Transformer Decoder): This is the part that actually writes the sentence. Because it now has the "Eyes" seeing the details and the "Guide" giving it the right cultural context, it can write a sentence that is both accurate and culturally rich.

3. The Secret Sauce: The "Double-Check" Loss Function

Usually, AI learns by trying to match its answer to a correct answer. But here, the authors added a special rule called Visual Semantic-Generation Fusion Loss.

Imagine a student taking a test with two parts:

1. Part A: Identify the main character (e.g., "This is a ghost").
2. Part B: Write a story about that character.

The teacher (the computer) grades both parts at the same time. If the student writes a great story but identifies the character wrong, they get a bad grade. If they identify the character right but write a boring story, they also get a bad grade.

This forces the AI to learn two things simultaneously: See the cultural symbols correctly AND Write about them beautifully.

4. The Result: A New Library of Stories

The researchers built a special library of 9,400+ Dongba paintings with descriptions to train their system. They taught the AI to recognize seven main themes, from "Hell Ghosts" to "Music and Dance."

When they tested it:

• Old AI models (like BLIP or ClipCap) were like tourists: They got the basic objects right but missed the meaning.
• The New AI (PVGF-DPC) was like a local historian. It didn't just say "There is a bat." It said, "This is a white bat, a divine messenger in Naxi mythology, riding a sacred eagle to the heavens to retrieve holy texts."

Why Does This Matter?

This isn't just about making AI smarter; it's about preserving culture. By teaching computers to understand the deep, symbolic language of Dongba art, we ensure that these ancient stories don't get lost or misunderstood in the digital age. The AI becomes a bridge, translating ancient visual wisdom into modern language for everyone to understand.

In short: They taught a computer to stop being a tourist and start being a local expert, using a "hint system" and a "double-check" rule to ensure the stories it tells are true to the culture.

Technical Summary

1. Problem Statement

The paper addresses the challenge of automatic image captioning for Dongba paintings, a unique pictorial art form of the Naxi people in southwestern China. While general image captioning has advanced significantly, applying state-of-the-art models to Dongba art faces two critical hurdles:

• Data Scarcity: There is a severe lack of large-scale, high-quality image-caption pairs specific to Dongba art, leading to overfitting when training deep learning models.
• Domain Shift & Cultural Nuance: Pre-trained models (trained on natural images) suffer from a domain shift. They fail to understand the specific iconography, symbolic meanings, and cultural context of Dongba paintings (e.g., distinguishing between a "deity," a "hell ghost," or specific ritual patterns), often resulting in hallucinations or culturally irrelevant, generic descriptions.
• Limitations of Existing Controllable Methods: Current controllable captioning (CIC) models rely on explicit text entities or keywords, which cannot capture the implicit cultural connotations and layered semantic depth inherent in Dongba art.

2. Methodology: PVGF-DPC Framework

The authors propose PVGF-DPC (Prompt and Visual Semantic-Generation Fusion-based Dongba Painting Captioning), an encoder-decoder framework designed to bridge the gap between generic natural-image captioning and culturally specific imagery.

A. Architecture Overview

The framework consists of three main components:

1. Visual Encoder (MobileNetV2):
   • Selected for its optimal balance between feature extraction quality and computational efficiency, crucial for the limited-data Dongba setting.
   • Extracts a 1280-dimensional discriminative visual feature vector ($e_x$) from resized (299×299) input images.
2. Content Prompt Module:
   • Maps the visual feature vector $e_x$ to a set of culture-aware content labels (e.g., "deity," "hell ghost," "flute playing") using a fully connected layer and softmax.
   • Constructs a post-prompt text (e.g., "This is a Dongba painting about [deity]...") based on the predicted label.
   • This prompt is concatenated with the caption text and fed into the decoder, providing explicit cultural context to steer generation.
3. Transformer Decoder:
   • A 10-layer Transformer decoder initialized with pre-trained BERT weights to enhance semantic understanding.
   • Visual-Textual Fusion: Unlike standard attention mechanisms, the image feature vector $e_x$ is injected directly into the Layer Normalization layers of the decoder. This allows visual semantics to continuously modulate the text generation process at every layer (similar to FiLM conditioning).

B. Visual Semantic-Generation Fusion Loss

To ensure the model learns both cultural relevance and visual accuracy, the authors introduce a joint optimization objective:
$$L_{fusion} = \alpha L_{text} + \lambda L_{prompt}$$

• $L_{text}$: Standard cross-entropy loss for caption generation.
• $L_{prompt}$: Cross-entropy loss for the content prompt module (predicting the thematic category).
• Effect: This forces the encoder to extract culturally salient visual features (to satisfy $L_{prompt}$) while the decoder generates fluent, semantically faithful descriptions (satisfying $L_{text}$).

3. Key Contributions

1. Dedicated Dataset: Creation of a Dongba Painting Captioning Dataset containing 9,408 augmented images with culturally grounded annotations. The data is categorized into seven thematic groups: deity/spirit, hell ghost/demon, bird/beast, flora, horseback riding/fishing, music/dance, and religious patterns.
2. Content Prompt Module: A novel mechanism that infers thematic categories from visual features and injects them as structured prompts, effectively mitigating hallucinations and improving thematic relevance.
3. Visual Semantic-Generation Fusion Loss: A composite loss function that jointly optimizes prompt prediction and caption generation, ensuring the model captures both visual details and cultural semantics.
4. State-of-the-Art Performance: Demonstration that the proposed framework significantly outperforms zero-shot and controllable baselines on this specialized domain.

4. Experimental Results

The model was evaluated on the constructed test set against nine baselines, including BLIP, ViECap, MacCap, ClipCap, and OFA.

• Quantitative Performance: PVGF-DPC achieved the highest scores across all seven metrics:
  • BLEU-1/2/3/4: 0.603 / 0.426 / 0.317 / 0.246
  • METEOR: 0.256
  • ROUGE: 0.403
  • CIDEr: 0.599
  • Notable: It outperformed the second-best model (ClipCap) by a significant margin in BLEU scores and surpassed ViECap by 0.416 in CIDEr.
• Ablation Studies:
  • Removing the prompt module (VGF-DPC) or the fusion loss (DBC) resulted in lower performance, confirming that both components provide complementary benefits.
  • The full model showed a 0.110 improvement in CIDEr over the variant without the prompt module, highlighting the prompt's role in enriching cultural specificity.
• Qualitative Analysis:
  • In case studies, PVGF-DPC correctly identified mythological roles (e.g., the white bat as a divine messenger) and specific iconography (e.g., "serpentine tail" for a Shu deity), whereas baselines like BLIP misidentified subjects or provided generic descriptions.

5. Significance and Impact

• Cultural Preservation: This work provides a technical pathway for the digital preservation and dissemination of Naxi Dongba culture, a form of intangible cultural heritage that has been underrepresented in AI research.
• Domain Adaptation Strategy: The paper demonstrates that prompt learning combined with semantic fusion is an effective strategy for adapting large-scale vision-language models to specialized, low-resource, and culturally dense domains.
• Mitigating Hallucination: By explicitly grounding the generation process in predicted cultural categories, the model significantly reduces the "hallucination" problem common in zero-shot captioning of artistic works.
• Future Directions: The authors suggest extending this framework to other cultural heritage domains and refining prompt designs to capture even finer-grained cultural attributes.