Technical Summary: GRANDGUARD

Problem Statement

As older adults increasingly adopt Large Language Model (LLM) chatbots for companionship and assistance, a critical safety gap has emerged. Existing safety benchmarks and moderation systems are primarily designed around "universal harms" (e.g., hate speech, adversarial attacks) and fail to account for Elderly-Specific Contextual Risks. These are scenarios where a prompt or instruction that is benign for a general or younger population becomes dangerous for older adults due to age-related vulnerabilities such as mobility limitations, cognitive decline, social isolation, and reduced digital literacy.

For instance, a request to "repair a ceiling light alone in the dark" is a standard DIY task for many but poses a severe fall risk for an older adult with limited mobility. Similarly, a query about "the highest tower nearby" is a benign tourism question unless paired with expressions of hopelessness, where it becomes a suicide risk. Current systems often miss these nuances because they lack the contextual sensitivity to recognize how age-specific factors transform a request into a hazard.

Methodology

The authors propose GRANDGUARD, a comprehensive framework comprising a taxonomy, a benchmark, and safeguards to assess and mitigate these specific risks.

1. Taxonomy Construction

The taxonomy was developed through a multi-source empirical investigation:

• Data Sources: Analysis of 25 AI incident reports, annotation of 1,000 posts from the r/eldercare community, and synthesis of six prior stakeholder studies (workshops and interviews with older adults, caregivers, and clinicians).
• Differentiation: A comparative annotation protocol was used to distinguish elderly-specific risks from general harms. A risk is retained only if it disproportionately affects older adults due to age-linked factors (e.g., frailty, cognitive decline), making the same interaction materially lower-risk for a younger adult.
• Structure: The resulting taxonomy is a three-level hierarchy containing 50 fine-grained risk types across five domains:
  • Mental Well-being: Neglect of care, self-harm, and unhealthy dependency on AI.
  • Financial: Direct manipulation, scams, and fraud susceptibility.
  • Medical: Dangerous advice and physical activity risks.
  • Toxicity: Manipulative control, sexual exploitation, ageism, and political exploitation.
  • Privacy: Sensitive data exposure and unauthorized device access.

2. Benchmark Construction

The authors constructed a benchmark of 10,404 labeled data points (6,498 prompts and 3,906 responses).

• Data Generation: 500 human-authored seed prompts (10 per third-level risk type) were expanded into 10,000 candidate prompts using LLMs. These were filtered by an LLM judge to ensure they exhibited subtle, elderly-specific risks without overt malice.
• Severity Weighting: An online study with 60 participants rated the severity of risk types on a 7-point Likert scale. The benchmark distribution was curated to reflect these severity weights, ensuring high-severity risks (e.g., self-harm) are adequately represented.
• Evaluation Criteria:
  • Prompt Evaluation: High-sensitivity rule; flagged as unsafe if an older adult is identifiable and the request matches a risk type without safeguards.
  • Response Evaluation: Dual criteria requiring Risk Indication (explicitly acknowledging the age-related vulnerability) and Harm Avoidance (refusing harmful enablement and offering safe alternatives).

3. Safeguard Development

Two complementary solutions were developed to address the identified safety failures:

• Fine-Tuned Detection: A Llama-Guard-3 model was fine-tuned using LoRA on a mixture of GRANDGUARD data and general safety datasets. This model is designed for high-recall detection in centralized deployments.
• Policy-Enhanced Moderation: An "Elderly-Sensitive Policy" was implemented for the gpt-oss-safeguard-20b model. This allows for structured, interpretable rules (e.g., flagging financial decisions for users with cognitive decline) without retraining, suitable for customizable care contexts.
• GRANDGUARD Agent: A lightweight agentic pipeline that uses the policy-enhanced safeguard to detect risks, analyze them, and prepend safety reasoning instructions to the user prompt before it reaches the downstream LLM.

Key Results

1. Baseline Model Performance

Evaluation of 10 leading LLMs (including GPT-5.1, Claude-Sonnet-4.5, Gemini-2.5) on the benchmark revealed widespread safety failures:

• High Failure Rates: Several leading models failed to properly address elderly-specific risks in over 50% of cases. For example, GPT-5.1 achieved only 56.2% safety, while smaller models like GPT-4.1-Mini scored as low as 22.6%.
• Knowledge–Action Gap: Self-diagnosis experiments revealed a significant gap between a model's ability to identify risks and its ability to act on that knowledge. While models could flag up to 95% of risky prompts when explicitly asked (Prompt Awareness), their actual response safety (Response Safety) was often much lower (e.g., a gap of +50.2 points for Gemini-2.5-Flash). This suggests models possess latent risk awareness but fail to translate it into safe generation behavior.

2. Safeguard Effectiveness

The proposed safeguards significantly improved safety metrics:

• Fine-Tuned Llama-Guard-3: Achieved 96.2% accuracy in prompt classification and 93.2% in response classification, a substantial improvement over the base model (63.3% accuracy).
• Policy-Enhanced gpt-oss-safeguard-20b: Improved prompt detection accuracy to 90.9% and response accuracy to 90.3% when augmented with the elderly-specific policy.
• GRANDGUARD Agent: When used as a context-augmentation layer, the agent improved response safety across all tested models. Notable gains included DeepSeek-V3.2 (from 39.6% to 91.8%) and Llama-4-Maverick (from 28.2% to 82.4%).

Significance and Claims

The paper claims to lay the groundwork for AI systems that move beyond general safety to specifically support aging populations. Its primary contributions are:

1. First Comprehensive Framework: It introduces the first taxonomy and benchmark specifically targeting elderly-specific contextual risks, addressing a gap where existing tools overlook age-linked vulnerabilities.
2. Empirical Evidence of Failure: It provides rigorous evidence that current state-of-the-art LLMs, despite strong general capabilities, systematically fail in elderly-specific contexts, often due to a "Knowledge–Action Gap."
3. Actionable Mitigation: It demonstrates that safety can be effectively improved not just by retraining models from scratch, but through targeted fine-tuning and policy-enhanced moderation that leverages existing model capabilities to recognize and mitigate contextual risks.

The authors emphasize that their work is intended for research and guardrail development, cautioning against the direct use of these benchmarks for real-world medical or financial decision-making without further validation. They also highlight the need to balance safety with user autonomy, advocating for "safer alternatives" rather than blanket refusals to avoid over-restriction of elderly users.