Trust, Safety, and Accuracy: Assessing LLMs for Routine Maternity Advice

1. Problem Statement

The paper addresses the critical public health challenge of maternal healthcare accessibility in India. Key issues include:

• Infrastructure & Cultural Barriers: Inadequate healthcare infrastructure, combined with social stigma, privacy concerns, and cultural taboos, often prevents women from consulting doctors regarding pregnancy-related issues.
• Rise of Digital Reliance: With over 830 million internet users in India, women are increasingly turning to Large Language Models (LLMs) as a first line of inquiry for health advice.
• The Risk: While digital access is expanding, there is a significant risk of misinformation and culturally insensitive responses. There is a lack of systematic evaluation regarding the accuracy, readability, and cultural alignment of leading LLMs (ChatGPT-4o, Perplexity AI, and GeminiAI) specifically within the Indian socio-cultural context.

2. Methodology

The authors employed a structured, two-phase comparative methodology to evaluate the performance of three LLMs against expert medical opinions.

A. Data Collection

• Query Design: 17 specific questions were formulated, covering the spectrum from early gestation to postnatal care. These included prevalent myths and misconceptions common in India.
• Phase 1 (LLM Generation): The LLMs (ChatGPT-4o, Perplexity AI, GeminiAI) were prompted to act as "local experienced obstetric and maternal care experts" to generate responses.
• Phase 2 (Expert Ground Truth): Corresponding answers were collected from qualified, experienced obstetricians and maternal care experts to serve as the gold standard for comparison.

B. Evaluation Framework

The study utilized three primary technical metrics to assess the LLM outputs:

1. Readability Assessment (Linguistic Metrics):
   To ensure accessibility for non-expert audiences (specifically women with diverse educational backgrounds), six standard metrics were applied:

   • Flesch Reading Ease (FRE): Higher scores indicate easier readability.
   • Flesch-Kincaid Grade Level (FKGL): Lower scores indicate text suitable for lower grade levels.
   • Gunning Fog Index (GFI): Estimates years of formal education required.
   • SMOG Index: Focuses on polysyllabic word count.
   • Dale–Chall Readability Score (DCRS): Measures familiarity of words.
   • Automated Readability Index (ARI): Uses character counts to estimate grade level.

2. Semantic Similarity (Cosine Similarity):

   • Used to measure the vector-based semantic alignment between LLM responses and expert responses.
   • Scores closer to 1 indicate high semantic similarity; scores near 0 indicate divergence.

3. Noun Entity Overlap (Jaccard Similarity):

   • Quantified the overlap of key noun entities (medical terms, conditions, anatomical references) between LLM outputs and expert answers.
   • A score of 1 implies complete conceptual overlap; 0 implies no shared terminology.

3. Key Contributions

• Context-Specific Evaluation: Unlike previous studies that may have focused on general medical accuracy, this paper specifically evaluates LLMs within the Indian socio-cultural context, addressing local myths and stigma.
• Multi-Metric Framework: The study integrates readability (accessibility), semantic similarity (meaning), and entity overlap (technical precision) to provide a holistic view of LLM performance in maternal health.
• Benchmarking Leading Models: It provides a comparative analysis of ChatGPT-4o, Perplexity AI, and GeminiAI, offering empirical data on which model is best suited for sensitive health communication in resource-limited settings.

4. Results

The study yielded distinct findings across the three evaluation dimensions:

• Readability:

  • ChatGPT-4o emerged as the superior model for accessibility. It achieved the highest FRE score (64.63) and the lowest FKGL (7.53), indicating its responses are understandable at a middle-school reading level (13–15 years old).
  • Perplexity AI and GeminiAI produced more complex text with higher grade-level requirements (e.g., Perplexity FKGL = 9.67), making them less accessible to the general public.
  • Conclusion: ChatGPT's language is clearer and better suited for public health communication.

• Semantic Similarity (Cosine Similarity):

  • Perplexity AI achieved the highest mean cosine similarity (0.206) across most expert responses (D1, D2, D3), suggesting its responses were semantically closest to the experts' meaning.
  • All models showed a significant drop in similarity with Expert D4, indicating a specific area of divergence in medical consensus or phrasing.

• Noun Entity Overlap (Jaccard Similarity):

  • ChatGPT demonstrated the strongest contextual alignment with a Jaccard score range of 0.172–0.216.
  • Perplexity AI was the most consistent model across experts D1, D2, and D3.
  • All models struggled to match the noun entities of Expert D4.

5. Significance and Conclusion

• Potential for Digital Health: The findings confirm that LLMs, particularly ChatGPT and Perplexity AI, have the potential to serve as supportive tools for maternal health in underserved Indian communities. They can effectively address myths and provide privacy-preserving, empathetic advice.
• Accessibility vs. Precision Trade-off: While Perplexity showed high semantic similarity, ChatGPT offered the best balance of clinical coherence and readability. For maternal health communication, readability is paramount; if the advice is too complex, it cannot be acted upon by the target demographic.
• Future Directions: The authors emphasize that while current results are promising, future work must include:
  • Domain-specific tuning of models using local health guidelines.
  • Expansion to diverse medical domains and socio-cultural contexts.
  • Continuous real-world validation to ensure safety and trust.

Final Verdict: The study concludes that ChatGPT-4o is currently the most effective tool for generating accessible, culturally sensitive, and clinically coherent maternity advice for the Indian population, though continuous monitoring is required to mitigate risks of misinformation.