Can Parents and Patients Understand Myopia Using Large Language Model-Based Chatbots?

This prospective comparative study found that AI chatbots, particularly ChatGPT and DeepSeek, provide reliable and high-quality information on myopia suitable for helping parents and patients understand the condition, while Gemini's responses were rated slightly lower but still adequate.

The Gist

Imagine you are a parent whose child has been diagnosed with myopia (nearsightedness). You are worried, you have a million questions, and you turn to the internet for answers. But the internet is like a giant, noisy marketplace: some stalls sell gold (accurate medical advice), while others sell painted rocks (misinformation).

Recently, a new kind of "market stall" has appeared: AI Chatbots (like ChatGPT, Gemini, and DeepSeek). They are like super-smart, fast-talking librarians who can read millions of books in a second. But the big question is: Can we trust these digital librarians to give us the right advice about our children's eyes?

This study set out to find the answer by putting these three AI librarians to a "taste test."

The Setup: A Blind Taste Test

The researchers acted like food critics. They didn't just ask random questions; they gathered 20 real questions that parents actually ask in doctor's offices (e.g., "Will my child need glasses forever?" or "Can we stop myopia from getting worse?").

They fed these questions to three different AI chatbots:

1. ChatGPT (The famous one from OpenAI)
2. Gemini (Google's AI)
3. DeepSeek (A newer, powerful AI)

The Twist: To keep it fair, the AI responses were hidden in a "blind box." The raters (four eye experts: two senior doctors, one junior doctor, and one researcher) didn't know which AI wrote which answer. They just saw the text and had to grade it on a scale of 1 to 5, like rating a movie:

• 1 (Very Poor): Dangerous or wrong info.
• 3 (Acceptable): Okay, but vague.
• 5 (Very Good): Perfect, clear, and accurate.

Note: The AI were told to keep answers short (under 50 words), like a quick text message, so parents wouldn't get overwhelmed.

The Results: Who Passed the Test?

Here is how the three "digital librarians" performed:

🏆 The Top Performers: ChatGPT and DeepSeek
These two were the star students. They consistently gave answers that the doctors rated as "Good" to "Very Good."

• Analogy: Think of them as a reliable GPS. If you ask, "How do I get to the hospital?" they give you a clear, accurate route. They explained complex eye science in simple language that parents could actually understand.
• The Score: About 66-67% of their answers were rated as high quality.

🥈 The Runner-Up: Gemini
Gemini did okay, but it stumbled a bit more often. Its answers were mostly "Acceptable," meaning they weren't dangerous, but they lacked the depth or clarity of the other two.

• Analogy: Think of Gemini as a GPS that gets you to the right city but sometimes suggests a route with a few potholes or confusing turns. It's not "broken," but you might need to double-check the directions.
• The Score: It had a higher rate of "Poor" answers (7.5%) compared to the others, and its overall rating was significantly lower.

The "Surprise" Factor: The Unpredictable Librarian

One of the most interesting findings was about consistency.

• The Experiment: The researchers asked the exact same question to the same AI on two different days.
• The Result: Sometimes, the AI gave a different answer!
• Analogy: Imagine asking your favorite chef, "How do you make pasta?" on Monday, and they give you a perfect recipe. But on Tuesday, they give you a completely different recipe with different ingredients. This happens because AI models are like living, breathing creatures that are constantly learning and changing, not static books.

What Does This Mean for You?

The Good News:
AI chatbots are becoming a powerful tool for parents. They can act as a "first aid kit" for information, helping you understand basic concepts about myopia quickly and for free. ChatGPT and DeepSeek, in particular, are doing a great job of translating "doctor-speak" into "parent-speak."

The Caution:
You shouldn't treat the AI like a doctor.

• The Metaphor: Think of the AI as a very knowledgeable tour guide. They can tell you about the history of the city (your eye condition) and point out the main sights. But they are not the architect who built the building. They cannot examine your specific child, look at their eyes, or prescribe a treatment plan.

The Bottom Line

If you are a parent with questions about myopia, these AI chatbots (especially ChatGPT and DeepSeek) are a great place to start. They can help you understand the basics and prepare you for a conversation with your real-life eye doctor.

However, just like you wouldn't trust a weather app to perform surgery, you shouldn't trust a chatbot to make medical decisions. Use the AI to learn, but always take your child to a real human expert for the final diagnosis and treatment.

Technical Summary

1. Problem Statement

Myopia is a rapidly growing global public health concern with significant vision-threatening complications and economic burdens. While clinical awareness is increasing, knowledge among parents and patients remains suboptimal. Patients increasingly rely on the internet and AI chatbots for health information, but the reliability of these sources varies. There is a critical need to evaluate whether Large Language Model (LLM)-based chatbots can provide accurate, evidence-based, and accessible information regarding myopia, its control, and its complications to laypersons.

2. Methodology

Study Design:
A prospective comparative reliability study conducted at the LV Prasad Eye Institute, India (February–March 2025). No human participants were recruited; therefore, ethical approval was waived.

AI Models Evaluated:
Three state-of-the-art LLMs were tested:

• ChatGPT (OpenAI GPT-5)
• Gemini (Google Gemini 2.0)
• DeepSeek (DeepSeek-R1 / V3)

Question Development (Stage 1):

• Source: 20 questions were generated based on real-world clinical interactions with parents of myopic children.
• Validation: Questions were validated through group discussions with clinicians and a survey of 25 parents.
• Categories:
  1. General questions on myopia (Q1–Q8).
  2. Myopia control and prevention (Q9–Q15).
  3. Complications of myopia (Q16–Q20).
• Constraint: To ensure readability for laypersons and reduce cognitive load, all chatbots were instructed to limit responses to 50 words.

Evaluation Protocol (Stage 2 & 3):

• Raters: Four masked raters evaluated the responses: two senior clinicians (>5 years experience), one junior clinician (3 years experience), and one researcher (4 years experience).
• Blinding: Raters were unaware of which chatbot generated which response (responses were randomized and labeled Chatbot-1, -2, -3).
• Scoring: A 5-point Likert scale was used:
  • 1: Very Poor (factually incorrect/misleading)
  • 2: Poor (major inaccuracies)
  • 3: Acceptable (partially correct/vague)
  • 4: Good (mostly accurate/clear)
  • 5: Very Good (accurate, clear, complete, accessible)
• Reliability Checks:
  • Repeatability: Responses were regenerated on different days with different login IDs to test consistency.
  • Intra-rater reliability: Raters scored the same responses twice (3-week interval).
  • Inter-rater agreement: Agreement between the four raters was calculated.

Statistical Analysis:

• Data analyzed using IBM SPSS Statistics 26.
• Metrics included Median (Q1, Q3), Cronbach's $\alpha$ (internal consistency), Intraclass Correlation Coefficient (ICC) for reliability, and Kruskal-Wallis/Mann-Whitney U tests for group comparisons.

3. Key Contributions

• Comparative Analysis: This is the first study to directly compare ChatGPT (GPT-5), Gemini (2.0), and DeepSeek specifically for myopia-related queries from a patient/parent perspective.
• Real-World Relevance: Questions were derived from actual clinical encounters and validated by parents, ensuring the study reflects genuine information needs.
• Rigorous Masking: The study employed a robust double-blind design where raters did not know the source of the AI responses, minimizing bias.
• Granular Scoring: The use of a 5-point Likert scale with specific exemplars allowed for nuanced evaluation of "good" vs. "acceptable" vs. "poor" quality.

4. Results

Overall Performance:

• The combined median rating for all chatbots was 4.0 ("Good").
• ChatGPT and DeepSeek performed significantly better than Gemini.
  • ChatGPT: Median 4.0 ("Good"). 66.0% of responses rated "Good" or "Very Good."
  • DeepSeek: Median 4.0 ("Good"). 67.5% of responses rated "Good" or "Very Good."
  • Gemini: Median 3.0 ("Acceptable"). Only 40.0% rated "Good" or "Very Good." It had the highest proportion of "Poor/Very Poor" ratings (7.5%).
• Statistical Significance: A significant difference was found among the three models ($p \le 0.001$). Pairwise comparison showed Gemini was significantly lower than the other two ($p \le 0.001$), while ChatGPT and DeepSeek showed no significant difference ($p=0.48$).

Category-Specific Findings:

• General Questions: ChatGPT and DeepSeek rated "Good"; Gemini rated "Acceptable."
• Control & Prevention: All three models performed equally well ("Good").
• Complications: ChatGPT rated "Good," while Gemini and DeepSeek were rated "Acceptable."

Reliability Metrics:

• Internal Consistency: Cronbach's $\alpha$ was 0.70 (acceptable).
• Repeatability: Responses varied when regenerated at different times (Cronbach's $\alpha$ was negative or low for ChatGPT and Gemini), indicating that LLM outputs are not deterministic and can fluctuate.
• Inter-rater Agreement:
  • ChatGPT: ICC 0.61 (Good agreement).
  • DeepSeek: ICC 0.57 (Good agreement).
  • Gemini: ICC 0.13 (Poor agreement), suggesting raters found Gemini's responses harder to evaluate consistently.

Specific Errors:

• Gemini frequently omitted the distinction between "low-dose" atropine and general atropine usage.
• ChatGPT occasionally provided poor responses regarding the reversibility of myopia, whereas others correctly noted it is controllable but not reversible.

5. Significance and Conclusion

Significance:
The study demonstrates that AI chatbots, particularly ChatGPT and DeepSeek, are viable tools for providing initial, high-quality education to parents and patients regarding myopia. They can effectively bridge the gap in health literacy, provided the information is verified. However, the variability in responses (repeatability issues) and the lower performance of Gemini highlight that these tools are not yet perfect substitutes for clinical advice.

Limitations:

• Word Limit: The 50-word constraint, while improving readability, may have sacrificed depth and nuance in complex medical explanations.
• Question Scope: Questions were curated by experts and may not capture the full diversity of unstructured patient queries.
• Model Volatility: The study noted that AI models are constantly evolving; results may differ as models are updated.

Conclusion:
AI chatbots can support patients and parents in understanding myopia, with ChatGPT and DeepSeek demonstrating superior reliability compared to Gemini. The authors recommend that these tools be used as supplementary educational resources, enhanced with evidence-based training and multilingual support, rather than as standalone diagnostic or treatment guides.