Evaluating Large Language Models' Responses to Sexual and Reproductive Health Queries in Nepali

This paper introduces the LLM Evaluation Framework (LEAF) to assess Large Language Models' responses to 14,000 Nepali sexual and reproductive health queries, revealing that only 35.1% of responses were "proper" due to significant gaps in accuracy, cultural appropriateness, and safety, thereby highlighting the urgent need for improved evaluation criteria in low-resource, sensitive domains.

The Gist

Imagine you have a very smart, well-read robot friend who knows a lot about the world. You can ask it anything, and it answers instantly, without judging you. This is what Large Language Models (LLMs) like ChatGPT are.

Now, imagine you are in Nepal, and you have a very personal, sensitive question about your body, your health, or your family planning. Maybe you are shy to ask a doctor, or there isn't one nearby. You turn to this robot friend.

This paper is like a report card for that robot friend, but specifically for how well it handles these tricky, personal health questions in the Nepali language.

Here is the story of what the researchers did, explained simply:

1. The Problem: The "Perfect" Answer isn't Enough

The researchers noticed that most people only check if a robot's answer is factually correct (like checking if a math answer is right). But for health questions, being "correct" isn't enough.

• Analogy: Imagine you ask a tour guide, "Where is the nearest hospital?"
  • If they say, "It's 5 miles north," that is accurate.
  • But if they say it in a language you don't understand, or they give you a map to a hospital that closed 10 years ago, or they say, "Just ignore the pain," that is dangerous.
  • The robot needs to be accurate, but also safe, culturally polite, and easy to understand.

2. The Solution: The "LEAF" Framework

The team invented a new way to grade the robots, which they called LEAF (LLM Evaluation Framework). Think of LEAF as a four-legged stool. If one leg is broken, the stool falls over. The four legs are:

1. Accuracy: Is the medical info right?
2. Language: Did it answer in Nepali (not English)?
3. Usability: Was the answer helpful, not too long, and culturally appropriate? (e.g., not suggesting a medicine you can't buy in Nepal).
4. Safety: Did it avoid giving dangerous advice or saying something offensive?

3. The Experiment: A Massive Chat Session

The researchers built a chatbot and invited 9,000 real people from all over Nepal to talk to it for about 30 minutes.

• Who asked? Regular people and Female Community Health Volunteers (local health workers).
• What did they ask? Over 14,000 questions about periods, pregnancy, contraception, and more.
• The Test: They asked the questions to two versions of the robot:
  • Robot A (ChatGPT-3.5): The standard, free version.
  • Robot B (ChatGPT-3.5 + RAG): The standard version, but with a "textbook" attached to it (Retrieval Augmented Generation) so it could look up facts from official Nepali health manuals.

4. The Results: The Robot is Smart, but Clumsy

After the chats, experts graded the answers using the LEAF framework. Here is what they found:

• The "Perfect" Score was Rare: Only 35% of the answers were "Proper." This means they were accurate, safe, in the right language, and helpful.
• Accuracy vs. Usefulness: The robot got the facts right 62% of the time. However, half of those "correct" answers still had problems.
  • Analogy: It's like a chef who cooks a delicious steak (accurate) but serves it on a plate that is too hot to touch (usability gap) or forgets to cut the meat (inadequate).
• The Biggest Issue: The robot often gave inadequate answers. It didn't give enough detail or missed key parts of the question.
• The Safety Issue: Surprisingly, the robot was mostly safe. Very few answers were offensive or dangerous (less than 1%). But the researchers warned that even one bad answer in health can be fatal.
• Language Trouble: Sometimes the user asked in Nepali, and the robot replied in English. Other times, it mixed languages weirdly.
• Robot B vs. Robot A: The robot with the "textbook" (RAG) was slightly better at facts, but still struggled with the nuances of the conversation.

5. The "GPT-4" Surprise

The researchers also tested a newer, smarter robot (GPT-4) on a small sample.

• Good News: GPT-4 was much better. It gave "Proper" answers 59% of the time (compared to 35% for the older one).
• Bad News: Even the super-smart GPT-4 sometimes got confused by "Romanized Nepali" (Nepali written with English letters, like "Kasto cha?"). It preferred the official script (Devanagari).

6. The Big Takeaway

The paper concludes that while AI is a promising tool for helping people in Nepal get health information anonymously, it is not ready to be a doctor yet.

• The Metaphor: Think of the current AI as a student nurse who has read all the textbooks but has never actually held a patient's hand. They know the theory (accuracy) but often forget the bedside manner (usability and safety).
• The Future: We need to train these robots better, especially on local culture and language, before we can fully trust them with sensitive health issues.

In short: The robot is smart, but it needs to learn how to be a good listener and a caring helper before it can replace a human health worker in Nepal.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) are increasingly integrated into daily life and healthcare, their evaluation in low-resource languages and culturally sensitive domains remains inadequate. Current evaluation methods predominantly focus on accuracy for objective queries in high-resource languages (like English), often neglecting usability (relevance, cultural appropriateness) and safety (confidentiality, sensitivity).

Specifically, in Nepal, barriers to Sexual and Reproductive Health (SRH) services include stigma, myths, and inadequate information access. While LLMs offer a potential anonymous, non-judgmental solution for SRH queries, there is a lack of robust frameworks to assess their performance in Nepali (a low-resource language) and within community settings rather than just clinical ones. Existing studies rarely evaluate the multidimensional gaps (usability and safety) that could render an accurate response harmful or unusable.

2. Methodology

A. The LEAF Framework

The authors introduced the LLM Evaluation Framework (LEAF), a multi-criteria, language-agnostic framework designed to assess LLM responses beyond simple accuracy. It comprises four major criteria:

1. Accuracy: Categorized as Accurate, Partially Accurate, or Inaccurate based on factual correctness.
2. Language: Assesses if the response matches the query language (Same, Different, or Mixed).
3. Usability Gaps: A multi-label classification including:
   • Relevance: Is the response contextually appropriate?
   • Adequacy: Does it contain all necessary components?
   • Cultural Appropriateness: Is it aligned with local context (e.g., availability of contraceptives in Nepal)?
   • Updatedness: Is the information current?
   • Length: Is it excessively long?
4. Safety Gaps: A multi-label classification including:
   • Safety: Does it risk user safety (e.g., advising silence on violence)?
   • Sensitivity: Is it offensive or biased?
   • Confidentiality: Does it inadvertently reveal personal information?

A response is defined as "Proper" only if it is accurate, in the same language, relevant, adequate, updated, concise, safe, non-offensive, and confidential.

B. Data Collection Platform

• LLM Selection: The study primarily used GPT-3.5 (ChatGPT1) and GPT-3.5 with Retrieval Augmented Generation (RAG) (ChatGPT2). The RAG system utilized an external knowledge base of 22 documents (booklets, leaflets, manuals) from Nepal's Ministry of Health.
• Open-Source Testing: Nine open-source models (e.g., Llama2, Mistral, BLOOM) were tested but rejected due to poor performance in Nepali comprehension and generation compared to GPT-3.5.
• Interface: A web-based chat interface was developed using Hugging Face's Text Generation Inference (TGI) server, PostgreSQL for storage, and Redis for caching.

C. Data Collection & Demographics

• Participants: 9,035 users recruited from October 2023 to February 2024 across 45 municipalities in all 7 provinces of Nepal.
  • Composition: 78.4% Community members (students, mothers, etc.) and 21.6% Female Community Health Volunteers (FCHVs).
  • Demographics: Predominantly young (21–30 years), female (63.4%), and mobile-device users (81.9%).
• Process: Users interacted with the chatbot for ~30 minutes in facilitated workshop settings.
• Dataset: From 10,535 initial conversations, 8,272 were valid. After filtering for non-GPT models and non-substantive queries, the final analysis focused on 14,132 meaningful Q&A pairs (27,689 total queries).

D. Evaluation Process

• Annotation: Four SRH experts manually annotated responses using the LEAF criteria via a dedicated labeling interface.
• Reliability: Inter-rater agreement was measured using Cohen's Kappa.
  • Accuracy (3-label): 0.764 (Substantial agreement).
  • Accuracy (Binary): 0.873 (Almost perfect agreement).

3. Key Results

A. Overall Performance

• "Proper" Conversations: Only 35.1% of responses met all LEAF criteria for a "proper" conversation.
• Accuracy: 62.1% of responses were factually accurate. However, 43.8% of these accurate responses still contained usability or safety gaps, proving that accuracy alone is insufficient for quality assurance.
• Inaccuracy: 13.8% of responses were inaccurate. Interestingly, the topics with the highest inaccuracy (menstruation, sex, contraceptives) were also the most frequently discussed, suggesting volume does not guarantee quality.

B. Usability Gaps

• Inadequacy: The most significant gap; 74.0% of responses were deemed inadequate (missing critical components).
• Irrelevance: 14.9% of responses were irrelevant to the query.
• Length: 10.4% were excessively long.
• Cultural/Outdated: Rare (<1%), but when present, they were predominantly associated with inaccurate responses.

C. Safety Gaps

Safety issues were less frequent but critical:

• Unsafe: 0.74% (98 responses) contained advice that could lead to unsafe actions (e.g., ignoring violence).
• Insensitive/Offensive: 0.38% (51 responses).
• Non-Confidential: 0.05% (7 responses) revealed identifiable info.

D. Language Performance

• Query Language: 84.2% of queries were in Nepali (Romanized or Devanagari).
• Response Language: Only 73.5% of responses matched the query language.
  • ~10.7% of Nepali queries received English responses.
  • 7.5% of responses were mixed-language (Nepali/English/Hindi).

E. GPT-3.5 vs. GPT-4 Comparison

A small-scale experiment (100 queries) compared GPT-3.5 and GPT-4:

• Accuracy: Improved from 26% (GPT-3.5) to 50% (GPT-4).
• Proper Conversations: Increased from 1 (GPT-3.5) to 12 (GPT-4).
• Usability: GPT-4 significantly reduced irrelevant and inadequate responses.
• Limitation: GPT-4 still struggled with Romanized Nepali, often producing errors when attempting to reply in Romanized script, though it performed better when switching to Devanagari.

4. Key Contributions

1. LEAF Framework: Introduced a novel, multi-criteria evaluation framework that moves beyond accuracy to include usability and safety, adaptable across languages and domains.
2. Low-Resource Language Focus: Applied rigorous evaluation to Nepali, highlighting specific challenges in low-resource language generation (script switching, cultural context).
3. Community-Based Evaluation: Shifted the evaluation context from clinical settings to community settings, involving diverse users (including FCHVs) in real-world scenarios.
4. Large-Scale SRH Dataset: Generated one of the largest datasets (14K+ queries) specifically for SRH queries in a low-resource setting, covering 32 distinct themes.

5. Significance and Limitations

Significance:

• The study demonstrates that while LLMs show promise for SRH in Nepal, current models (even GPT-3.5/4) are not yet reliable enough for standalone deployment due to significant usability and safety gaps.
• It underscores that accuracy is necessary but not sufficient; a medically accurate answer that is culturally inappropriate or unsafe is dangerous.
• The framework provides a blueprint for developers and policymakers to improve LLMs for sensitive health topics in the Global South.

Limitations:

• Facilitation Bias: Data was collected in workshop settings with facilitators, which may have influenced user questioning (e.g., peer pressure, facilitator prompting).
• Demographic Gaps: Underrepresentation of the elderly, very young adolescents, and rural populations due to device access and digital literacy.
• Token Limits: GPT-3.5 responses were frequently cut off mid-sentence due to token limits, affecting the evaluation of length and completeness.
• Resource Intensity: The manual labeling of 30K+ responses by experts was time-consuming, highlighting the need for automated evaluation tools in the future.

Conclusion:
The paper concludes that LLMs require significant strengthening in adequacy, cultural alignment, and safety before they can be trusted as primary SRH counselors in Nepal. The LEAF framework offers a critical pathway for iterative improvement and responsible AI deployment in sensitive domains.