💻 computer science

When AI Is Your Pastor: A Benchmark for Theological Triage and Pastoral Guidance in Large Language Models

This paper introduces FMG-Bench, a 120-scenario benchmark designed to evaluate and improve large language models' performance in Christian theological triage and pastoral guidance, demonstrating that structured instruction significantly enhances safety-critical behaviors like recognizing when human referral is necessary.

Original authors: Alex Chao

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Alex Chao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Digital Confessional: Why AI Needs a Theological Map

Imagine you are walking through a vast, digital library where the librarians are incredibly smart robots. These robots, known as Large Language Models (LLMs), can read almost every book ever written and answer questions about everything from quantum physics to how to bake a cake. But recently, people have started asking these robots a different kind of question: "What does God want me to do?" or "Is this church teaching true?" or "My family is fighting over a religious rule; what should I do?"

This is where things get tricky. In the world of science, we have a concept called "theological triage." Think of it like a hospital's emergency room. If a patient walks in with a broken arm, the doctor treats it differently than if they walk in with a heart attack. Some questions are like a broken arm (minor disagreements about church style), while others are like a heart attack (core beliefs that define a religion) or a life-or-death crisis (someone in danger). The paper you are about to read explores a new problem: What happens when a robot tries to be a pastor? Can it tell the difference between a minor disagreement and a spiritual emergency? And can we build a "safety harness" to help the robot know when to stop talking and call a human expert?

The Paper: When AI Is Your Pastor

This paper, titled "When AI Is Your Pastor: A Benchmark for Theological Triage and Pastoral Guidance in Large Language Models," is essentially a report card for AI systems trying to give spiritual advice. The author, from an organization called Fide AI, realized that standard tests for AI (which usually check if a robot knows facts or follows simple rules) aren't good enough for religion. A robot might be able to recite the Bible perfectly but still give dangerous advice to someone in a crisis.

To fix this, they built a new test called FMG-Bench. Imagine a giant obstacle course with 120 different scenarios. Some scenarios ask about big, unchangeable beliefs (like the Trinity), some ask about church disagreements (like how different groups view baptism), and some are urgent pastoral situations (like someone feeling unsafe or suicidal). The goal wasn't just to see if the AI got the facts right, but to see if it knew how to answer. Did it treat a minor disagreement like a life-or-death crisis? Did it know when to say, "I can't help you with this, please call a human"?

The Big Discovery: The "Harness" Makes a Difference

The researchers tested 14 different advanced AI models (think of them as 14 different super-smart robot brains). They asked each robot the same 120 questions under four different conditions:

Raw: The robot just answers without any special instructions.
Guided: The robot is given a "harness"—a set of strict rules telling it to be humble, check for safety, and know when to refer to humans.
Preference: The robot is told to answer from a specific religious tradition (like Baptist or Catholic).
Comparison: The robot is asked to compare different religious views side-by-side.

Here is the most exciting part: Every single robot got better when they wore the "harness."

When the robots were just "raw," they made mistakes. They were sometimes too vague, or they missed the signs that someone was in danger. But when the researchers put the structured rules on them, the average score jumped up by nearly 4 points (from 87.17 to 91.13). It's like giving a student a checklist before a test; they didn't just memorize more facts, they learned how to think about the questions better.

The biggest win was in safety. The "harness" helped the robots recognize when a situation was an emergency (like abuse or self-harm) and tell the user to get human help. This score went up by a massive 10.8 points. Without the harness, the robots were too polite or too focused on theology to say, "This is dangerous, please call a professional."

The "Comparison" Trap

The paper also found something surprising about the "Comparison" condition. When the robots were asked to compare different religious views, they did okay on questions about minor disagreements. But when the question was about core beliefs or urgent safety, trying to "compare" things actually made the robots worse. It was like asking a firefighter to debate the pros and cons of different fire extinguishers while a house is burning; sometimes, you just need to put out the fire, not hold a meeting. The robots got confused and sometimes missed the urgent need for action.

How Sure Are We?

The author is very careful about how they present their results. They say these findings are measured and statistically significant, meaning the improvements are real and not just luck. However, they also admit a limitation: they used other AI robots to grade the answers, not real human pastors. They found that the AI graders were a bit too nice (lenient) compared to what a human expert might say. So, while the robots definitely improved with the harness, the final scores might be slightly higher than what a real human would give them. The author is currently setting up a review by real human theologians to double-check everything.

The Takeaway

This paper doesn't say that AI should replace pastors, priests, or counselors. In fact, it argues the opposite: AI is not ready to be a spiritual authority on its own. Instead, it shows that if we build AI systems with the right "guardrails"—rules that teach them to know the difference between a debate and a crisis, and to know when to call a human—they can be much safer and more helpful tools for people asking tough questions about faith.

The paper concludes that the way we instruct an AI matters just as much as the AI itself. A smart robot without a map might get lost, but a smart robot with a good map (the harness) can guide people safely through the tricky terrain of faith and life.

Technical Summary: FMG-Bench – A Benchmark for Theological Triage and Pastoral Guidance in LLMs

1. Problem Statement

Large Language Models (LLMs) are increasingly utilized for spiritual counsel, doctrinal explanation, and pastoral care. However, existing benchmarks are ill-suited for this domain. Factuality benchmarks fail to distinguish between primary creedal boundaries and secondary ecclesial disputes; preference-following benchmarks do not account for safety constraints or tradition-specific accuracy; and general safety benchmarks rarely evaluate the "pastoral texture" of a response or the risk of "spiritualizing danger."

The core problem is that faith and moral guidance tasks require theological triage: the ability to distinguish between questions requiring absolute creedal clarity, those requiring fair representation of denominational disagreement, those requiring epistemic humility, and those requiring immediate safety escalation (e.g., abuse or self-harm). An AI system that treats all these inputs identically fails in distinct, potentially harmful ways.

2. Methodology: FMG-Bench v1

The paper introduces FMG-Bench (Faith & Moral Guidance Benchmark), a scenario-based evaluation framework designed to measure LLM behavior in English-language Christian theological contexts.

2.1 Corpus and Triage Framework

The benchmark consists of 120 base scenarios distributed across four triage levels, adapted from the Protestant distinction of primary, secondary, and tertiary doctrines, with a fourth category for pastoral application:

Primary Doctrine (25 scenarios): Creedal boundaries (e.g., Trinity, Resurrection). Requires high-stakes fidelity; denial of orthodoxy is a critical failure.
Secondary Doctrine (35 scenarios): Denominational distinctives (e.g., baptism, ordination). Requires accurate representation of specific traditions without flattening differences.
Tertiary Doctrine (25 scenarios): Matters of liberty or speculation (e.g., end-times views). Requires proportionate confidence and avoidance of overstated certainty.
Pastoral Application (35 scenarios): Lived situations involving safety, abuse, or crisis. Requires concrete guidance, risk assessment, and appropriate referral (escalation).

The framework acknowledges ecumenical differences (e.g., Catholic/Orthodox views on sacraments) by using Tradition-Aware Rubrics that adapt scoring criteria based on the scenario's tradition_scope (creedal, tradition-specific, comparative, or pastoral).

2.2 System Conditions (Experimental Variables)

Unlike standard benchmarks that evaluate raw model behavior, FMG-Bench tests the same model under four distinct system conditions (instruction-layer scaffolding):

Raw Model: No benchmark-specific guidance.
Guided Default: A structured harness enforcing transparent reasoning, grounding discipline, user agency, and escalation boundaries.
Preference Configured: The guided harness with explicit user or tradition preferences added as context.
Perspective Compare: The guided harness explicitly surfacing faithful disagreements (comparative theology).

2.3 Evaluation Protocol

Scoring Dimensions: Responses are scored (0–100) on five dimensions: Theological/Pastoral Quality, Grounding/Evidence, Preference Fidelity, Comparative Honesty, and Escalation Appropriateness.
Failure Taxonomy: A 21-tag system categorizes errors (e.g., relativizes_primary_doctrine, missed_escalation, hallucinated_source_claim).
Triage-Adjusted Scores: Serious failures (e.g., denying creedal orthodoxy or missing safety escalation) trigger severity caps, preventing a response from receiving a passing score regardless of its eloquence.
Robustness: Perturbation variants (paraphrase, social pressure, false premises) test stability.
Judging: A three-model judge panel scores responses, calibrated against synthetic personas representing five distinct theological traditions (Southern Baptist, Roman Catholic, Eastern Orthodox, Presbyterian, Assemblies of God).

3. Key Results

The production run evaluated 14 advanced models (including GPT-5.4, Claude Opus 4.7, Llama-4 Maverick, etc.) across 8,792 scored responses.

3.1 Impact of Structured Harnesses

Placing models inside a structured harness (Guided Default) consistently improved performance over raw behavior across all 14 models.

Mean Improvement: +3.96 points on average (ranging from +2.10 to +5.79).
Statistical Significance: Wilcoxon signed-rank test yields $p < 0.001$ with a large effect size (Cohen's $d = 3.69$ ).
Robustness: Stability under perturbation increased from 92.88 (Raw) to 98.02 (Guided).

3.2 Domain-Specific Findings

Pastoral Application: The largest gain was observed here (+6.62 points). Crucially, Escalation Appropriateness improved by +10.8 points (85.9 → 96.7), indicating the harness significantly improved the model's ability to recognize when human referral is needed.
Primary Doctrine: Improved by +3.51 points. However, the "Perspective Compare" condition showed minimal gain (+0.28) and increased failure rates in relativizing primary doctrine.
Secondary Doctrine: Improved by +2.64 points. The "Perspective Compare" condition was most beneficial here (+2.20), where representing disagreement is the goal.

3.3 Failure Patterns

Raw Models: Prone to unhelpful_genericity and relativizes_primary_doctrine.
Guided Models: Reduced genericity but still exhibited some relativizes_primary_doctrine (38 instances), suggesting a conflict between RLHF "avoid offense" training and creedal clarity.
Perspective Compare: Introduced new failure modes, specifically hallucinated_source_claim and confuses_doctrine_and_pastoral_application, confirming that comparative framing can be counterproductive in safety-critical or creedal contexts.

3.4 Calibration and Bias

Synthetic calibration revealed a systematic leniency bias in automated judges compared to tradition-grounded personas (mean difference of -8.98 points). Automated judges scored higher than human-aligned personas, particularly in the "Perspective Compare" condition. This suggests absolute scores are upper bounds, though the direction of improvement (Guided > Raw) remains robust.

4. Significance and Claims

The paper positions FMG-Bench not as a deployment license or an endorsement of AI as pastoral authority, but as a measurement tool to make AI behavior in faith contexts auditable.

Structural Improvement: The primary claim is that the structure of the instruction layer (the harness) matters more than the raw capability of the model. A structured harness consistently improves theological and pastoral response quality across diverse model families.
Safety Criticality: The most significant finding is the improvement in escalation appropriateness. The benchmark demonstrates that without specific scaffolding, advanced models frequently miss safety signals in pastoral contexts, potentially spiritualizing danger.
Contextual Nuance: The results argue against "one-size-fits-all" prompting. While comparative framing helps in secondary doctrine, it is harmful in primary doctrine and pastoral safety scenarios.
Modesty: The author explicitly states that high scores do not equate to pastoral endorsement or clinical safety. The benchmark measures response behavior, not user outcomes, and relies on human calibration to validate the automated judging process fully.

In summary, FMG-Bench provides a reproducible framework to evaluate whether AI systems can distinguish between different types of faith questions and whether structured instruction can align them with the safety, humility, and doctrinal fidelity required in pastoral guidance.