AI and Authenticity in Islamic Research: A Critical Evaluation of Generative AI Reliability, Hallucination, and Source Fidelity in Quranic, Hadith, and Fiqh Knowledge

Technical Summary: AI and Authenticity in Islamic Research

Problem Statement
The rapid adoption of generative AI (reaching ~16.3% of the global population and 92% of higher education students by 2025) has introduced a critical reliability gap in high-trust religious domains. While AI fluency is high, empirical evidence regarding its ability to provide authentic, verifiable, and trustworthy Islamic knowledge is limited. Islamic scholarship relies heavily on authenticated primary sources (Qur'an and Hadith), precise attribution, recognition of legitimate scholarly disagreement (ikhtilaf), and appropriate handling of uncertainty. Current benchmarks indicate that even frontier models struggle with factual grounding (e.g., 61.7%–83.6% accuracy on factuality benchmarks) and citation fidelity (up to 19.9% fabricated citations in literature reviews). In the Islamic domain, existing benchmarks like IslamicMMLU show significant performance variance (39.8%–93.8% accuracy) and a lack of comprehensive evaluation regarding open-ended, real-world user interactions, particularly concerning source verification, Madhhab (school of thought) sensitivity, and the risk of hallucinated religious rulings.

Methodology
This study employs a mixed-method comparative evaluation framework to assess six leading generative AI systems (ChatGPT, Gemini, Claude, Copilot, DeepSeek, and Dola AI). The research design prioritizes real-world usage over controlled laboratory benchmarks:

• Dataset: A survey instrument comprising 50 realistic, open-ended Islamic questions was developed, categorized into 10 sets covering Qur'anic interpretation, Hadith explanation, Fiqh (jurisprudence), ethics, pastoral guidance, and Madhhab-sensitive topics.
• Data Collection: Responses were collected from participants in Australia and the United Kingdom. Participants were given unrestricted freedom to select any publicly available AI platform, model, or subscription level to simulate authentic information-seeking behavior.
• Analysis Dimensions: Responses were analyzed using both qualitative and quantitative metrics, including:
  • Accuracy & Authenticity: Domain-specific correctness across Qur'an, Hadith, and Fiqh.
  • Citation Fidelity: Verification of source attribution, presence of fabricated references, and completeness of Hadith citations (book, chapter, number).
  • Hallucination Analysis: Identification of unverifiable religious claims and missing scholarly attribution.
  • Jurisprudential Consistency: Evaluation of how systems handle Madhhab diversity, scholarly disagreement, and uncertainty recognition.
  • Geographical Variability: Comparison of response consistency between Australian and UK datasets to assess the influence of regional routing or retrieval pipelines.

Key Findings

1. Domain-Dependent Reliability: AI performance is highly correlated with the complexity of the knowledge domain. Systems achieved the highest reliability in Qur'anic interpretation (4.4/5) and ethical guidance (4.0/5), where broad scholarly consensus exists. Performance dropped significantly in Hadith explanation (3.0/5) and was lowest in Fiqh reasoning (2.1/5) and Madhhab sensitivity (~2.3/5).
2. Citation Deficiencies and Hallucinations: Recurring issues were identified across all systems, including:
   • Missing exact Hadith numbers and book references.
   • Generalized attributions (e.g., "Sahih Muslim" without specific citation).
   • Fabricated or unverifiable religious claims.
   • Overconfidence in presenting disputed rulings as definitive facts.
3. Jurisprudential Handling: Significant variance exists in how models handle legal reasoning. Gemini and Claude demonstrated superior performance in recognizing scholarly disagreement, maintaining internal consistency, and acknowledging uncertainty. In contrast, systems like DeepSeek and ChatGPT often provided simplified, overly confident rulings or failed to distinguish between competing Madhhab opinions.
4. Geographical and Stylistic Variability: While semantic content was generally consistent between Australian and UK datasets, notable differences emerged in supporting references, citation completeness, and explanatory depth. Australian responses often contained more detailed Fiqh analysis, suggesting potential influences from regional retrieval pipelines or user prompting styles.
5. Behavioral Fingerprints: Distinct stylistic patterns were observed (e.g., Claude's philosophical/academic tone vs. ChatGPT's conversational summary style), indicating that while source material overlaps, model architectures produce different interpretive framings.

Key Contributions

• Empirical Evaluation: This study provides one of the first comprehensive empirical evaluations of generative AI reliability specifically within Islamic knowledge domains, moving beyond multiple-choice benchmarks to open-ended, real-world queries.
• Risk Assessment: It establishes a risk framework indicating that while AI is suitable for introductory learning and information discovery, it carries high risk (scores approaching 4.7/5) when used for independent religious rulings or complex Fiqh reasoning without human oversight.
• Citation Analysis: The research highlights a critical gap in citation fidelity, demonstrating that current systems frequently fail to provide the traceable evidence required for authentic Islamic scholarship.
• Geographical Insight: It identifies that AI-generated religious responses are not entirely deterministic and can vary based on geographical access conditions and retrieval environments.

Significance and Claims
The paper concludes that current generative AI systems should be regarded as assistive technologies for introductory Islamic learning and information discovery, but not as authoritative sources for religious rulings, scholarly research, or high-trust guidance. The authors assert that the "fluency" of AI responses does not equate to "authenticity." The study emphasizes that without verification against authenticated primary sources and qualified scholarly expertise, reliance on AI for religious guidance risks amplifying misunderstandings and providing misleading rulings. The findings offer practical guidance for researchers, educators, AI developers, and the Muslim community to navigate the integration of AI into religious contexts responsibly, prioritizing source verification and scholarly oversight.