SACRED: A Faithful Annotated Multimedia Multimodal Multilingual Dataset for Classifying Connectedness Types in Online Spirituality

1. Problem Statement

• Research Gap: Spirituality is a fundamental aspect of human experience that transcends culture, yet social science research on online spirituality is hindered by a lack of high-quality, large-scale datasets. Existing studies often rely on limited, non-digital data.
• Complexity of the Domain: Spirituality is abstract, deeply personal, and evolving. Defining it is challenging as it encompasses both theistic and non-theistic dimensions.
• Multimodal Challenge: Online spirituality is expressed through both text (posts, answers) and visuals (images, memes). Current computational methods struggle to integrate these modalities, and manual annotation of large multimodal datasets is prohibitively expensive and time-consuming.
• LLM Reliability: While Large Language Models (LLMs) are powerful, their ability to classify abstract spiritual concepts and adhere to strict theoretical definitions without hallucination remains untested in this specific domain.

2. Methodology

A. Dataset Construction (SACRED)

The authors created SACRED, the first annotated multimodal, multilingual dataset for online spirituality.

• Data Sources: Data was scraped from three major platforms using Python scripts:
  • Reddit: 10 spirituality-related subreddits (e.g., r/Buddhism, r/Meditation).
  • Instagram: 17 hashtags (e.g., #spirituality, #meditation), collecting images and captions.
  • Quora: 12 topics related to spirituality and religion.
• Total Volume: 6,769 Instagram images, 3,819 Quora answers, and 4,922 Reddit posts.
• Annotation Process:
  • Annotators: Five experts with backgrounds in Computational Linguistics and social sciences.
  • Guidelines: A strict definition of spirituality was used: "The pursuit and practice of experiences and beliefs that influence and nurture the spirit, fostering personal growth, meaning, and a sense of connection to something greater than oneself."
  • Classification Task: Annotators first determined if content was spirituality-related. If yes, they classified it into one of four Connectedness Types:
    1. Connectedness to Self: Authenticity, inner peace, self-knowledge.
    2. Connectedness to Others: Compassion, community, gratitude.
    3. Connectedness to Nature: Relationship with the natural world.
    4. Connectedness to Transcendence: Connection to God, the universe, or a higher power.
  • Multilingual Handling: Annotators recorded the language of captions and text within images (e.g., memes).

B. Evaluation Framework

The study evaluated the performance of 13 models (11 LLMs and 2 traditional approaches) on the SACRED dataset.

• Models Tested:
  • Traditional: Rule-based keyword dictionaries and a Fine-tuned BERT model.
  • LLMs (Text/Multimodal): LLaMA3.1, Reka, GPT-4o/4o-mini/4.1-mini, DeepSeek-V3, Qwen2.5-VL, LLaVA-1.5, Claude Sonnet-4.5, Gemini-2.5-Flash, Gemma3.
• Experimental Settings:
  • Zero-shot: Models classified without explicit definitions.
  • Definition-injected: Models were provided with the strict definitions of the four connectedness types to guide classification.
• Metrics: Accuracy, Precision, Recall, and F1-score.

3. Key Contributions

1. SACRED Dataset: The creation of the first faith-annotated, multimodal, multilingual dataset specifically for online spirituality research.
2. Discovery of a New Category: Through the analysis of "unclassifiable" posts, the study identified a fifth type of connectedness: "Connectedness to Art." This category captures spiritual experiences derived from creative expression (e.g., art, music, literature) that do not fit neatly into the other four categories.
3. Comprehensive LLM Benchmark: A rigorous evaluation of state-of-the-art models on abstract spiritual concepts, comparing text-only, vision-only, and multimodal capabilities.
4. Theoretical Insight: Demonstrated that providing strict definitions significantly improves model performance, highlighting the importance of contextual grounding in computational social science.

4. Key Results

A. Model Performance

• Text Classification:
  • DeepSeek-V3 achieved the highest accuracy (79.19% on the Quora test set) in the definition-injected setting.
  • Definition Injection: Providing definitions consistently improved performance across models (e.g., LLaMA3.1 improved from 45.27% to 54.05% on Instagram captions).
  • Traditional vs. LLM: Advanced LLMs significantly outperformed rule-based methods and fine-tuned BERT, indicating that contextual understanding is crucial for this domain.
• Visual Classification (Images):
  • GPT-4o-mini achieved the highest F1-score (63.99%) for classifying spiritual connectedness in images.
  • Qwen2.5-VL (7B parameters) surprisingly outperformed larger models like LLaVA-1.5 (13B) and Gemma3 (12B), achieving 68.31% accuracy.
  • Error Analysis: Models struggled most with Quora answers due to their length and the presence of multiple connectedness types within a single response, leading to confusion between local context and the dominant theme.

B. Data Statistics

• Platform Differences: Quora had the highest proportion of spirituality-related content (53.94%), while Reddit had the lowest (17.76%).
• Language Diversity: The dataset is multilingual, with significant non-English content in German, Spanish, and Chinese, though English remains dominant.

5. Significance and Limitations

Significance

• Computational Spirituality: This work bridges the gap between theology/social science and NLP, providing a foundational resource for studying how spirituality is evolving in the digital age.
• Methodological Advancement: It proves that LLMs can effectively handle abstract, multi-modal classification tasks when guided by strict theoretical frameworks.
• New Theoretical Insight: The identification of "Connectedness to Art" suggests that current theoretical models of spirituality may need expansion to account for creative expression as a spiritual conduit.

Limitations

• Language Bias: Despite being multilingual, the dataset is predominantly English. Non-English languages (German, Spanish) are underrepresented, limiting global applicability.
• Model Hallucination: Even with explicit instructions, some models (e.g., GPT series) occasionally hallucinated labels or failed to strictly adhere to the five-category constraint.
• Reasoning Disabled: The study did not enable "reasoning" modes (like Chain-of-Thought) in the models to control variables, potentially underestimating the maximum capability of advanced models like DeepSeek-R1 or GPT-5.

Conclusion

The SACRED dataset and the accompanying evaluation demonstrate that while current LLMs are powerful tools for analyzing online spirituality, they require careful prompt engineering (definition injection) and human oversight. The study successfully quantifies the performance of various models, identifies a new dimension of spiritual connectedness, and sets a benchmark for future research in computational religion and health communication.