GeMi: A Graph-based, Multimodal Recommendation System for Narrative Scroll Paintings

This paper presents GeMi, a novel Graph Neural Network-based multimodal recommendation system designed to preserve and promote the endangered art of narrative scroll paintings by leveraging text and image data to suggest similar works for viewing and purchase.

The Gist

Imagine a world where ancient storytellers, known as "Singing Painters," travel from village to village. They carry massive, hand-painted scrolls that tell epic tales of gods, monsters, and daily life. As they unroll the painting, they sing a song that matches the pictures, explaining the story to the crowd.

Unfortunately, this beautiful art form is dying out. The scrolls are fragile, the songs are being forgotten, and there are very few painters left. To save them, researchers collected thousands of photos of these scrolls and recorded the songs. But now they faced a new problem: How do you help people find the specific stories they love in this massive, messy digital archive?

Enter GeMi (Graph-based, Multimodal Recommendation System). Think of GeMi as a super-smart, digital "Art Librarian" designed specifically to save and share this endangered art.

Here is how GeMi works, explained through simple analogies:

1. The Problem: A Messy Library

Imagine you walk into a library where the books are a mix of torn pages, blurry photos, and handwritten notes in different languages. Some books have pictures but no text; others have text but no pictures.

• The Challenge: A normal computer recommendation system (like the one Netflix uses) is used to clean, organized data. It would get confused by this messy, "folk art" data. It wouldn't know that a picture of a tiger in one scroll is related to a song about a tiger in another scroll if the text descriptions are vague or missing.

2. The Solution: The "Super-Translator" (LLMs & Vision Models)

Before GeMi can recommend anything, it needs to understand what it's looking at.

• The Analogy: Imagine you have a translator who is also an art critic.
  • The Text Cleaner: First, GeMi uses a "Smart Translator" (a Large Language Model) to rewrite the messy, old handwritten song lyrics into clear, modern English. It fixes spelling errors and summarizes the story so the computer understands the meaning, not just the words.
  • The Eye and Ear: Then, it uses a "Super Eye" (Vision-Language Model) to look at the painting and listen to the song description simultaneously. It learns that a picture of a "mythical bird" and a song about a "flying creature" are actually talking about the same thing, even if the words are different.

3. The Brain: The "Social Network of Art" (Graph Neural Networks)

Once GeMi understands the content, it needs to figure out how to connect the dots. This is where the Graph comes in.

• The Analogy: Imagine a giant spiderweb.
  • Each node (dot) on the web is a single panel of a scroll painting.
  • The strings connecting them represent relationships.
  • If two paintings both feature a "Tree," a string connects them. If a user liked a painting with a "Mythical Hero," the web learns to suggest other paintings with heroes.
  • The Magic: Unlike a simple list, this web allows information to flow. If you like a painting with a tiger, the web "passes a message" to all the paintings connected to tigers, telling them, "Hey, this user might like you too!" This helps GeMi find hidden connections that a simple search engine would miss.

4. Handling the "Missing Pieces" (Dealing with Noise)

The data collected from the field was imperfect. Sometimes a song was recorded but the painting was lost; sometimes the text was damaged.

• The Analogy: Imagine trying to solve a puzzle where some pieces are missing or blurry.
  • GeMi uses a special technique called a Variational Autoencoder. Think of this as a "Guessing Game" engine. If a piece of the puzzle (a text description) is missing, GeMi looks at the picture and the surrounding puzzle pieces to probabilistically guess what the missing text probably said. It doesn't just ignore the missing data; it fills in the gaps with its best educated guess, allowing the recommendation to keep working.

5. Learning Your Taste (User Preferences)

Finally, GeMi learns what you like.

• The Analogy: Imagine a personal shopper who watches what you pick up in a store.
  • GeMi creates a "User Profile" based on the concepts you enjoy (e.g., "I love stories about trees" or "I prefer myths about gods").
  • It then scans the giant spiderweb of art. If you love "Trees," it doesn't just look for the word "Tree"; it looks for the vibe of trees in the paintings and songs, suggesting the most relevant scrolls for you to view or buy.

Why Does This Matter?

• Saving Culture: It acts as a digital time capsule, ensuring that even if the physical scrolls rot away, the stories and art survive in a format people can actually find and enjoy.
• Helping Artists: By making it easy for people to find and buy these scrolls online, it gives money directly to the struggling artists who keep the tradition alive.
• A New Kind of AI: This paper shows that we can build AI that doesn't just work on perfect data (like Amazon products) but can handle the messy, beautiful, and incomplete reality of human culture.

In short: GeMi is a high-tech, cultural detective that cleans up old stories, connects them on a giant digital web, and uses that web to introduce you to the ancient art you didn't know you were looking for.

Technical Summary

1. Problem Statement

The paper addresses the challenge of preserving and recommending narrative scroll paintings, a traditional performing art form from Eastern India (specifically Bengal) that is currently endangered. These artworks are unique because they combine visual imagery (scroll panels) with oral traditions (songs/poems) to tell stories.

The core technical challenges identified are:

• Data Scarcity and Sparsity: There is a lack of digitized, structured data for these art forms. Existing datasets are small, and user-item interaction data (clicks, ratings) is non-existent or extremely sparse.
• Multimodal Misalignment: In field-collected data, images and text (lyrics) are often not perfectly aligned, or one modality may be missing entirely.
• Noise and Ambiguity: Raw textual descriptions are noisy, stylistically heterogeneous, and semantically under-specified. Visual features can be ambiguous (e.g., background trees vs. main subjects).
• Cold Start: Traditional collaborative filtering fails due to the lack of historical user interaction data.
• Class Imbalance: Specific semantic concepts (e.g., "Tree" vs. "Mythological Character") appear with vastly different frequencies in the dataset, leading to biased model training.

2. Methodology: The GeMi System

The authors propose GeMi, a content-based recommendation system augmented with user preferences, built on a Graph Neural Network (GNN) architecture. The system is designed to handle multimodal data (text and images) and learn latent graph structures without relying on dense user interaction logs.

A. Data Collection and Preprocessing

• Dataset: A custom dataset was created from field surveys in West Bengal, India, containing 189 panels from 28 scrolls paired with 13 songs.
• Annotation: Panels were annotated for the presence/absence of three key concepts: Animals, Trees, and Mythological Characters.
• Synthetic User Generation: Due to the lack of real user data, the authors generated synthetic user profiles using Monte-Carlo sampling. They created preference vectors based on the semantic labels of sampled items to simulate user behavior for evaluation.

B. Multimodal Feature Extraction (The "GeMi" Pipeline)

The system employs a sophisticated three-stage feature extraction process:

1. LLM-Based Semantic Canonicalization:
   • Raw text descriptions are processed using TinyLlama (a Large Language Model).
   • The LLM acts as a deterministic rewriting function to normalize noisy, heterogeneous text into compact, concept-centric representations, reducing lexical variance before fusion.
2. Contrastive Vision-Language Pretraining (SigCLIP):
   • A Sigmoid-Contrastive Language Image Pretraining (SigCLIP) model is used. Unlike standard Softmax-CLIP, SigCLIP treats image-text pairing as independent binary classification problems, which is robust to batch size variations and class imbalances.
   • It learns a joint embedding space where semantically aligned image-text pairs are geometrically proximate.
3. Multimodal Variational Autoencoder (VAE):
   • A Product-of-Experts (PoE) VAE is used to probabilistically fuse visual and linguistic information.
   • It models uncertainty by learning a latent variable $z$ that generates both modalities. The fused posterior mean serves as a deterministic feature representation, allowing the system to handle missing modalities or noisy data effectively.

C. Graph Structure Learning (GSL)

The system constructs a graph where nodes represent scroll panels.

• Graph Construction: Edges are formed based on symmetric k-Nearest Neighbor (kNN) connectivity using cosine similarity of the multimodal embeddings.
• Handling Imbalance: To address the sparsity of the "Tree" label, the authors employ Minority-Aware Edge Augmentation, adding extra edges specifically between nodes labeled as "Tree" to ensure sufficient information flow for minority classes.
• Graph Architectures: Three GNN variants are evaluated:
  1. GeMi-GCN: Standard Graph Convolutional Network (supervised).
  2. GeMi-GAE: Graph Auto-Encoder (unsupervised reconstruction objective).
  3. GeMi-VGAE: Variational Graph Auto-Encoder (probabilistic latent space, explicitly modeling uncertainty).
• Training Paradigms: The system is tested in both Transductive (test nodes known during training) and Inductive (test nodes unseen) settings to simulate real-world deployment where new scrolls are added.

D. Recommendation Mechanism

• Label-Conditioned Relevance: Recommendations are framed as a semantic consistency problem. A user's inferred preference for a label (e.g., "Mythology") is matched against the ground-truth labels of candidate panels.
• Plug-and-Play CF: Collaborative filtering is treated as a modular component, allowing user preferences to be integrated via synthetic profiles.

3. Key Contributions

1. Novel Dataset: Creation of the first curated, multimodal dataset for narrative scroll paintings, including aligned images, lyrics, and semantic annotations.
2. GeMi Architecture: A novel recommendation framework that integrates LLM-based text canonicalization, SigCLIP, and Multimodal VAEs with Graph Structure Learning.
3. Handling Sparsity and Noise: Introduction of specific techniques (Minority-Aware Edge Augmentation, Sigmoid-Contrastive loss, and VAE uncertainty modeling) to tackle the unique challenges of small, noisy, and imbalanced cultural heritage datasets.
4. Comprehensive Benchmarking: Rigorous comparison against state-of-the-art homogeneous and heterogeneous graph-based recommenders (e.g., LATTICE, PMGT, GraphSAGE) in both transductive and inductive settings.

4. Empirical Results

The system was evaluated using Precision@K on synthetic user profiles across the three semantic labels.

• Performance vs. Baselines:
  • GeMi consistently outperformed state-of-the-art baselines (both transductive and inductive).
  • GeMi-GAE achieved the highest score for Mythology (0.78 Precision) in transductive settings with LLM-VAE features.
  • GeMi-VGAE showed superior performance in capturing uncertainty, particularly for the Mythology label in inductive settings (0.70).
  • GeMi significantly improved performance on the sparse Tree label compared to baselines, which struggled due to class imbalance.
• Impact of LLMs:
  • Using LLM-enhanced features (TinyLlama) significantly improved performance over base features (raw text + SigCLIP/VAE). The LLM's ability to canonicalize text was crucial for abstract concepts like "Mythology."
• Transductive vs. Inductive:
  • Transductive learning generally yielded more stable and higher performance, especially for semantically complex labels.
  • Inductive learning was competitive, proving the system's ability to generalize to new, unseen scrolls, though performance varied by label and graph type.
• Ablation Studies:
  • Focal Loss: Moderate focusing parameters ($\gamma \approx 2.0$) were optimal for handling class imbalance.
  • Edge Dropout: Moderate dropout rates (0.1–0.2) improved robustness by preventing overfitting to noisy edges.
  • kNN Parameter: The optimal $k$ was non-monotonic and label-dependent, highlighting the need for careful hyperparameter tuning based on feature geometry.

5. Significance and Future Work

• Cultural Preservation: GeMi provides a scalable technical solution for archiving and promoting endangered performing arts, making them accessible to a global audience.
• Technical Innovation: The paper demonstrates that combining Generative AI (LLMs/VAEs) with Graph Learning is highly effective for small-scale, multimodal datasets where traditional collaborative filtering fails.
• Future Directions:
  • Integration of Generative AI (Diffusion Models) to create missing visual content for songs that lack illustrations.
  • Development of dynamic pricing mechanisms to support the economic sustainability of the artists.
  • Deployment of the system via UPFAR (User Preferences for Folk Art Recommendation), an online tool currently collecting real-world user data to further refine the model.

In conclusion, GeMi represents a significant step forward in applying advanced AI to digital humanities, successfully bridging the gap between multimodal data processing, graph learning, and the specific needs of cultural heritage conservation.