ReLay: Personalized LLM-Generated Plain-Language Summaries for Better Understanding, but at What Cost?

1. Problem Statement

Plain Language Summaries (PLS) are designed to make scientific research accessible to lay audiences, particularly in healthcare. However, current PLS are typically "one-size-fits-all," failing to account for individual differences in:

• Background knowledge: A novice patient needs different explanations than an experienced one.
• Information needs: Users have varying goals (e.g., prevention vs. treatment management).
• Comprehension levels: Misunderstanding health information can lead to poor real-world decisions.

While Large Language Models (LLMs) offer the potential to personalize these summaries using user data, it remains unclear:

1. Whether personalization actually improves comprehension and perceived quality.
2. Which personalization strategies (e.g., metadata vs. narrative vs. retrieval) are most effective.
3. How to balance the benefits of personalization against safety risks, such as hallucinations and the reinforcement of user biases.

2. Methodology: The RELAY Benchmark

The authors introduce RELAY, a human-centered benchmark and dataset comprising 300 participant–PLS pairs from 50 lay participants.

A. Participant Recruitment & Characteristics

• Cohort: 50 US-based participants with no medical background, diverse educational levels (from high school to PhD), and varying health literacy.
• Data Collected: Demographics, health information needs, topic familiarity, trust in sources, and AI usage patterns.
• Design: Each participant reviewed 6 scientific abstracts (from Cochrane systematic reviews) across two settings:
  1. Static Setting: Participants read an expert-written PLS and answered comprehension/quality questions.
  2. Interactive Setting: Participants conversed with a chatbot (GPT-5.2) to ask questions, received an LLM-generated personalized PLS, and answered the same questions.

B. Experimental Setup

• Models Evaluated: Five LLMs spanning proprietary and open-source architectures:
  • Proprietary: GPT-4o, GPT-5.2.
  • Biomedical Specialized: MedGemma-27B.
  • Open-Weight: Qwen3-4B-Instruct, Mistral-7B-Instruct-v0.3.
• Personalization Strategies:
  1. Zero-Shot Baseline: No user context.
  2. Profile-Based Prompting:
     • Metadata: Structured user attributes (e.g., education, health literacy).
     • Backstory: A first-person narrative generated from the metadata.
  3. Retrieval-Augmented Generation (RAG):
     • Within-User: Retrieving similar past abstracts/questions from the same user.
     • Cross-User: Retrieving abstracts from users with similar profiles.

C. Evaluation Metrics

The study uses a dual-axis evaluation framework: Personalization and Safety.

• Personalization Metrics:
  • Readability: Alignment between the summary's reading level and the user's target level (measured via Flesch-Kincaid, etc.).
  • Style: Cosine similarity of stylistic features (lexicon, emotion) between summary and user reference texts.
  • Knowledge Alignment: Proportion of user-requested supports (definitions, background) or questions answered.
• Safety Metrics:
  • Hallucination:
    • Faithfulness ($F_{faith}$): Are simplification claims supported by the source abstract?
    • Factuality ($F_{fact}$): Are explanation claims verified against PubMed evidence?
  • Bias Reinforcement: Does the summary introduce or amplify biased framing regarding protected groups?

3. Key Results

A. Effectiveness of Personalization

• Comprehension & Quality: Interactive (personalized) settings significantly outperformed static (expert-written) settings.
  • Mean comprehension scores increased from 9.03 (static) to 9.83 (interactive).
  • Perceived quality ratings improved across all 9 dimensions, with the largest gains in Explanation (+0.45), Importance (+0.38), and Tailoring (+0.35).
• User Variability: The benefits of personalization did not significantly correlate with specific demographic or health literacy variables, suggesting a broad but non-uniform improvement.

B. Best Personalization Strategies

• Profile-Based vs. RAG: Profile-based prompting (using user metadata or backstory) generally outperformed RAG approaches across readability, style, and knowledge alignment.
• Metadata vs. Backstory:
  • Readability: Narrative backstories often improved readability alignment for open-source models.
  • Knowledge Alignment: Structured metadata yielded the best results for proprietary models (e.g., GPT-5.2 achieved $KN_{static}=0.554$ with metadata vs. $0.525$ with backstory).
• Model Performance:
  • Open-Source (Qwen, MedGemma): Excelled at readability and style alignment.
  • Proprietary (GPT-5.2): Excelled at knowledge alignment and safety (faithfulness).

C. The Safety-Effectiveness Trade-off

The study identified a fundamental tension between personalization and safety:

• Non-personalized Baselines: Achieved the highest safety scores (lowest hallucination and zero bias reinforcement) but performed poorly on personalization metrics (high readability mismatch, low knowledge alignment).
• Personalized Methods: Improved user alignment but introduced risks.
  • Bias Reinforcement: Narrative backstories were the only condition where bias reinforcement was observed (e.g., MedGemma-27B with backstory had a bias rate of 0.7%, Qwen3-4B had 6.5%). This suggests narrative framing may cause models to validate user beliefs rather than critically assess them.
  • Hallucination: Personalized summaries showed slightly higher rates of factual errors compared to non-personalized baselines, particularly in open-source models.

4. Key Contributions

1. RELAY Dataset: A novel, human-centered benchmark containing 300 participant–PLS pairs with rich annotations (term familiarity, interaction logs, comprehension scores, and quality ratings) covering both static and interactive settings.
2. Comprehensive Evaluation: A systematic comparison of five LLMs across multiple personalization strategies, evaluating both effectiveness (comprehension, style, knowledge) and safety (hallucination, bias).
3. Empirical Insights:
   • Personalization improves lay understanding and perceived quality.
   • Direct profile conditioning (metadata/backstory) is more effective than retrieval-based personalization.
   • The Trade-off: Maximizing personalization often compromises safety (increasing hallucination and bias risk), while prioritizing safety limits personalization effectiveness.

5. Significance and Implications

• Health Communication: The findings suggest that while LLMs can transform health communication from passive reading to active, personalized engagement, developers must carefully manage the trade-off between relevance and reliability.
• Model Selection: For high-stakes health applications, proprietary models (like GPT-5.2) may be preferable for safety and knowledge alignment, while open-source models may be better suited for stylistic adaptation if safety guardrails are strengthened.
• Future Directions: The paper advocates for "trustworthy personalization" methods that can adapt to user needs without reinforcing biases or introducing medical hallucinations. It highlights the need for evaluation frameworks that treat safety and personalization as coupled, rather than independent, objectives.

In conclusion, RELAY demonstrates that personalized PLS are superior to static ones for lay audiences but require robust safety mechanisms to prevent the introduction of medical misinformation and bias.