Technical Summary: Omissive Bias in Religious Representation

Problem Statement

As Large Language Models (LLMs) increasingly serve as primary sources of guidance for personal, moral, and existential questions, there is a critical need to evaluate whether these systems reflect the diverse value frameworks of their users. While existing research extensively documents presence-based biases (e.g., stereotypes, political leanings, toxic content), this paper identifies a distinct form of representational harm: omissive bias.

The authors define omissive bias as the systematic failure of aligned LLMs to surface perspectives that are substantively relevant to a query, specifically religious frameworks. Given that 75–80% of the global population identifies with a religion, and many rely on religious resources for navigating grief, forgiveness, relationships, and moral dilemmas, the consistent absence of religious references in LLM responses to everyday ethical questions constitutes a form of "symbolic annihilation." The paper posits that current alignment protocols, which often default to a Western, secular-rationalist baseline, may inadvertently erase religious frameworks from the discourse, even when those frameworks are highly relevant to the user's context.

Methodology

The AllFaith Religious Representation Benchmark

To operationalize omissive bias, the authors constructed the AllFaith Religious Representation Benchmark, consisting of 150 open-ended questions.

• Question Sourcing: The dataset was derived from two primary sources:
  1. WildChat-1M: A large corpus of real user-LLM interactions. An automated pipeline extracted candidate questions, which were then filtered via LLM classification and manual curation to ensure they were genuine, high-stakes ethical questions (e.g., about grief, marriage, honesty) that did not explicitly mention religion but where religious perspectives would be a valuable part of a strong answer.
  2. Faith-Community Contributions: Supplemental questions were curated by partners from various faith traditions (Jewish, LDS, Protestant, Catholic, etc.) to ensure diversity and mitigate selection bias inherent in the WildChat data.
• Question Characteristics: The questions cover themes such as "Inner Life" (depression, addiction), "Relationships" (forgiveness, marriage), and "Worldview & Ethics" (meaning, truth). Crucially, the questions are not about religion itself; they are personal dilemmas where religion is one of several potential resources.

Evaluation Protocol

• Model Testing: The benchmark was evaluated against 27 frontier and open-source models (including GPT-5 variants, Claude 4.6/4.7, Gemini 3.1, Llama 4, Grok 4, and others).
• LLM-as-Judge Rubric: Responses were evaluated using an LLM-as-judge framework (specifically Gemini 3.1 Pro Preview). The rubric was deliberately set with a low bar:
  • Score 0: No religious representation.
  • Score 1–4: Increasing levels of religious mention, practice, or leadership.
  • Metric: For the primary analysis, scores were binarized (0 = no religion, 1 = any religious mention).
• Human Baseline: To establish a ground truth for "expectation," the authors conducted a survey with 1,125 US participants (n=11,250 ratings). Participants rated the 343 initial questions on a 1–5 scale regarding whether they would expect an answer to contain religious references. The final 150-question benchmark was selected based on the questions with the most significant mismatch between human expectations and model behavior.

Key Results

Systematic Underrepresentation

Across all 27 models tested, LLMs consistently underrepresented religion relative to human expectations.

• Overall Performance: The mean binarized rating for religious relevance across all models was 0.084, indicating that in roughly 91.6% of cases, models provided answers with zero religious content.
• Human vs. Model Gap: Human expectations were significantly higher, with a mean rating of approximately 0.63 (on a binarized scale derived from the 1–5 survey) for the same questions. The correlation between human expectations and model behavior was weak ($r = 0.257$ to $0.441$ depending on the metric), suggesting models do not align with user expectations regarding religious relevance.

Asymmetric Omission

The omission of religious frameworks was not uniform across topics:

• Abstract vs. Practical: Models were more likely to invoke religion for abstract existential questions (e.g., "What happens after death?", "What is the meaning of life?") where secular answers might feel insufficient.
• Practical Dilemmas: Models rarely invoked religion for practical, interpersonal situations (e.g., "How do I cope with grief?", "How to handle infidelity?", "Forgiving a spouse"). In these domains, where religion historically provides significant communal and ritual support, models defaulted to secular, therapeutic, or procedural advice (e.g., suggesting therapy or self-reflection rather than prayer, confession, or clergy).

Model Variance

While the trend of underrepresentation was consistent, there was variance among models.

• Outliers: The Grok-4.20 model showed a notably higher propensity for religious mentions compared to other models. The authors hypothesize this may be due to Grok's training data, which includes real-time social media discourse (X) where religious debates occur more frequently, or a different alignment strategy.
• General Trend: Most models, including flagship models from OpenAI, Anthropic, and Google, clustered around a mean score of 0.1, suggesting a uniform "safe" alignment that avoids religious framing in daily ethical reasoning.

Significance and Claims

The paper makes a modest but significant claim: Current LLMs overlook critical opportunities to reflect religious frameworks that many people draw on when navigating personal and ethical challenges.

• Not Anti-Religious Bias: The authors explicitly state they are not adjudicating which values LLMs should hold, nor do they claim the omission is necessarily "anti-religious bias" in a malicious sense. Instead, they argue it is likely an emergent property of alignment incentives, safety policies, and training data that favor secular, therapeutic, or procedural advice.
• Representational Harm: Drawing on communication theory (Gerbner & Gross) and algorithmic fairness (Crawford; Barocas et al.), the paper argues that the absence of a perspective held by the majority of the global population constitutes a representational harm. By failing to surface religious resources in contexts where they are culturally and personally relevant, LLMs risk "erasing" these frameworks from the digital discourse shaping society.
• Design Tension: The authors highlight a design tension: LLM providers may fear that over-representing religion feels like proselytizing, yet under-representing it erases a vital resource for billions. They argue for a strategy that explicitly handles religion with clear policies, allowing models to acknowledge religious resources as a valid option without imposing them.

Conclusion

The AllFaith Religious Representation Benchmark provides the first formal operationalization of omissive bias as a primary LLM evaluation paradigm. The results demonstrate that while LLMs are capable of discussing religion in abstract terms, they systematically fail to integrate religious frameworks into practical, everyday ethical decision-making, diverging significantly from human expectations. The authors conclude that for LLMs to faithfully reflect the pluralistic moral lives of humans, they must be better calibrated to recognize when religious perspectives are contextually appropriate, optional, or irrelevant.