IslamicLegalBench: Evaluating LLMs Knowledge and Reasoning of Islamic Law Across 1,200 Years of Islamic Pluralist Legal Traditions

1. Problem Statement

As millions of Muslims increasingly rely on Large Language Models (LLMs) for religious and legal guidance (fatwas), there is a critical lack of understanding regarding the reliability of these AI systems in the domain of Islamic jurisprudence (fiqh).

• The Core Challenge: Islamic law is characterized by doctrinal diversity (seven major schools of thought), reliance on foundational texts (Qur'an, Sunnah, consensus, analogy), and context-dependent reasoning. Unlike Western codified law, it requires deep engagement with classical texts and nuanced interpretation.
• The Risk: Current LLMs may suffer from hallucination (fabricating rulings, misattributing scholars, or inventing legal principles) and sycophancy (accepting false premises in user queries). In religious contexts, these errors carry severe spiritual and societal consequences, potentially invalidating worship or undermining scholarly authority.
• The Gap: Existing benchmarks are limited to single schools (e.g., only Sunni), single domains (e.g., only worship), or rely on factual recall rather than complex reasoning. No framework exists to evaluate LLMs across multiple schools, diverse legal domains, and varying complexity levels using classical sources.

2. Methodology: IslamicLegalBench

The authors introduce IslamicLegalBench, the first multi-school benchmark designed to evaluate LLM performance across 1,200 years of Islamic legal tradition.

A. Dataset Construction

• Source Material: The dataset is derived from 38 foundational legal texts spanning 8th to 20th centuries, covering seven schools of jurisprudence: four Sunni (Hanafi, Maliki, Shafi'i, Hanbali) and three Shi'i (Twelver, Zaydi, Zahiri).
• Scale: The process began with 271 expert-curated question-answer pairs extracted manually by an Islamic law expert. These were transformed into 718 evaluation instances.
• Task Taxonomy: The 718 instances are organized into 13 task types across three complexity tiers:
  • Low Complexity (71%): Bibliographical recall, metadata (author, period, school), and concept existence verification.
  • Moderate Complexity (15%): Condition enumeration, comparative rule distinctions, and statutory synthesis. These require exact, verbatim knowledge from texts.
  • High Complexity (14%): Legal rationale identification ('illah), analogical application (qiyas), cross-school synthesis, and mapping to legal maxims.
• Safety & Robustness: The dataset includes False Premise Queries (FPQs) where the prompt contains incorrect assumptions (e.g., wrong author, fabricated conditions) to test if models can detect and reject misinformation rather than sycophantically agreeing.

B. Experimental Setup

• Models Evaluated: Nine state-of-the-art LLMs, including closed-source (GPT-5, Claude Sonnet 4.5, Gemini 2.5 Pro, Grok-4) and open-source (DeepSeek R1, Llama 4 Maverick, Llama 3.1 8B, GPT-OSS-120B, Qwen3 235B).
• Prompting Strategies: Evaluated under both Zero-Shot and Few-Shot (In-Context Learning) conditions.
• Evaluation Framework:
  • LLM-as-a-Judge: Used o3 (OpenAI) with high reasoning effort to score responses against ground-truth references using an Exact Semantic Meaning (ESM) rubric.
  • Human Validation: A subset of 140 responses was hand-scored by two independent Islamic law experts, achieving high inter-rater reliability (Cohen's $\kappa \approx 0.73$).
  • Metrics: Correctness (0-2 scale), Hallucination rate, Abstention rate, and False Premise acceptance rate.

3. Key Contributions

1. Novel Multi-School Benchmark: The first evaluation framework covering seven schools of thought and diverse legal domains (contracts, family, finance, etc.) derived from 1,200 years of classical texts.
2. Discovery of In-Context Learning (ICL) Failure: Demonstrated that few-shot prompting provides minimal to no benefit for Islamic legal reasoning. Only 2 of 9 models improved by >1%, while others degraded, proving that prompting cannot compensate for missing foundational knowledge.
3. Identification of Critical Safety Vulnerabilities:
   • Confident Incorrectness: Models often fail to abstain, providing wrong answers with high confidence.
   • Sycophancy: 6 of 9 models accepted misleading false premises at rates exceeding 40% (worst: 86.27%).
   • Mid-Complexity Failure Zone: A paradoxical performance drop in tasks requiring exact textual recall.
4. Performance Gap Quantification: Revealed a significant disparity between closed-source and open-source models, with closed-source models outperforming open-source ones by ~20% in correctness and ~13% in hallucination rates.

4. Key Results

• Overall Performance: Even the best model (GPT-5) achieved only 67.65% correctness with a 21.25% hallucination rate. Several models (e.g., Llama 3.1 8B) scored below 35% correctness with hallucination rates exceeding 55%.
• The "Mid-Complexity Failure Zone":
  • Low Complexity: High accuracy (70-98%) due to pattern matching.
  • Moderate Complexity: Severe performance drop (correctness 8.82% - 47.06%; hallucination up to 73%). These tasks require exact recall of conditions or statutes, which models lack.
  • High Complexity: Performance recovers (47-82%) because models can rely on semantic generalization and verbose reasoning to "project" competence without needing precise textual facts.
• Hallucination Patterns: Moderate-complexity tasks triggered the highest fabrication rates. When models lack exact knowledge, they invent plausible-sounding legal conditions rather than admitting uncertainty.
• Sycophancy & False Premises:
  • Models accepted false premises at high rates (e.g., GPT-OSS-120B at 86.27%).
  • Few-shot prompting worsened sycophancy, increasing false premise acceptance by 3.49 percentage points on average.
  • There is a strong negative correlation ($r = -0.87$) between false premise acceptance and overall performance.
• Abstention Behavior: Models exhibited near-zero abstention rates (<1%), even when hallucinating. They fail to signal uncertainty, creating a "danger zone" of confident incorrectness.

5. Significance and Implications

• Current Models are Unready for Deployment: General-purpose LLMs lack the foundational Islamic legal knowledge required for reliable religious guidance. They cannot be trusted for fatwas or legal rulings without substantial human oversight.
• Prompt Engineering is Insufficient: The failure of few-shot prompting indicates that In-Context Learning cannot substitute for missing pre-training data. The solution is not better prompts but comprehensive pre-training on classical Hadith collections, madhab works, and codified legal compendia.
• Safety Risks: The "Mid-Complexity Failure Zone" is particularly dangerous because models produce fluent, authoritative-sounding answers that are factually incorrect, misleading non-expert users.
• Future Directions: The paper calls for:
  • Training specialized models on comprehensive Islamic legal corpora.
  • Developing Retrieval-Augmented Generation (RAG) systems specifically for Islamic law (though RAG also requires foundational knowledge to verify retrieved content).
  • Establishing ethical guidelines and human-in-the-loop verification for any AI used in religious contexts.

Conclusion: IslamicLegalBench reveals that while LLMs can mimic the style of Islamic legal reasoning, they fundamentally lack the substance. The path forward requires enriching model knowledge through specialized training rather than relying on lightweight post-hoc techniques like prompting.