Evaluation of symptom checker formats to support health literacy and trust in AI: Results from an online randomised-controlled trial

In a randomized controlled trial of 2,110 Australian adults, AI-enhanced symptom checker formats significantly improved immediate symptom management knowledge and reduced unnecessary primary care intentions for self-careable conditions compared to standard formats, without negatively impacting trust or acceptability, though knowledge gains were not sustained after two weeks.

The Gist

Imagine you wake up with a fever and you've thrown up a couple of times. You're feeling shaky, worried about dehydration, and you don't know if you should stay in bed, call a doctor, or rush to the emergency room. Most of us would pull out our phones and ask a "symptom checker"—a digital tool that acts like a friendly, knowledgeable nurse on the internet.

But here's the problem: Not all symptom checkers are created equal. Some give you a generic, one-size-fits-all answer, while others might use fancy new technology (Artificial Intelligence) to give you a personalized, tailored plan.

This study asked a simple question: Does using a "smarter," AI-enhanced symptom checker make us trust it more, understand our health better, or actually follow its advice?

The Experiment: A Digital Taste Test

The researchers set up a massive online experiment with over 2,000 Australians. They created a fake scenario (fever and vomiting) and showed participants a screenshot of a symptom checker.

Think of this like a restaurant tasting menu. Everyone ordered the same "dish" (the health scenario), but they were served five different "versions" of the advice:

1. The Standard Dish: The original, existing advice from the national health service (Healthdirect). It was clear but a bit generic.
2. The AI-Enhanced Dish: The same advice, but rewritten by AI to be more specific to your symptoms, explaining why you should stay home, and admitting, "Hey, an AI helped write this."
3. The Step-by-Step Dish: The AI version, but with numbered instructions (Step 1, Step 2, Step 3) to make it easier to follow.
4. The Multimedia Dish: The AI version, but with a video and pictures to help you visualize what's happening.
5. The "Behind the Scenes" Dish: The AI version, but with a long, detailed explanation of exactly how the AI works.

They tested this in two situations:

• Scenario A (Low Acuity): "You're sick, but you can probably stay home and rest."
• Scenario B (Moderate Acuity): "You're sick, and you should see a doctor within 24 hours."

What Did They Find?

1. The "Stay Home" Superpower

When the advice was to stay home, the people who saw the AI-enhanced versions were much less likely to say, "I'm going to the doctor anyway."

• The Analogy: Imagine a GPS telling you, "There's a traffic jam ahead, take the back road." The old GPS just said, "Traffic ahead." The new GPS explained why the back road is better and gave you a clear map. People trusted the new GPS enough to actually take the back road instead of panicking and trying to force their way through the jam (going to the doctor unnecessarily).
• Result: The tailored, AI advice helped people feel confident enough to manage their illness at home, potentially saving the healthcare system from a flood of unnecessary visits.

2. The "Knowledge Boost" (That Faded Like a Sunset)

Immediately after reading the AI-enhanced advice, people knew much more about how to handle their fever and vomiting. They understood what to do if they got worse.

• The Analogy: It was like drinking a strong cup of coffee. Right after drinking it, you were wide awake and sharp.
• The Catch: Two weeks later, that "coffee" had worn off. The knowledge boost didn't stick. People forgot the specific details, regardless of which format they saw.

3. The "AI Fear" Myth

A lot of people worry that if a machine writes the advice, they won't trust it. They might think, "Is a robot trying to kill me?"

• The Result: Nope. The study found that explicitly telling people, "This was written with AI," did not make them trust the tool less. They trusted the advice just as much as the standard version.
• The Analogy: It's like finding out your favorite chef used a new, high-tech oven. As long as the food tastes good and the chef is reputable, you don't care about the oven. The trust came from the source (the national health service), not the tool used to write the message.

4. The "See a Doctor" Scenario

When the advice was to see a doctor, the fancy AI formats didn't change much. People were already planning to go to the doctor, so the extra bells and whistles didn't make them stay home or leave earlier. They just followed the advice to go.

The Big Takeaway

This study is like a proof of concept for the future of health advice.

• Good News: We can use AI to make health advice clearer, more personal, and less scary. It helps people feel confident enough to stay home when they don't need a doctor, which is great for everyone.
• The "Human" Element: Telling people "AI did this" doesn't scare them away. In fact, if the advice is good and comes from a trusted source, people are happy to accept it.
• The Challenge: Just because people learn something quickly doesn't mean they remember it later. We need to figure out how to make that knowledge stick, perhaps by making the tools interactive (like a real conversation) rather than just static pictures.

In short: AI-enhanced symptom checkers are a promising tool to help us make smarter health decisions, reduce unnecessary trips to the doctor, and build trust without losing our humanity. They are the "smart GPS" for our health, and we are ready to use them.

Technical Summary

Title

Evaluation of symptom checker formats to support health literacy and trust in AI: Results from an online randomised-controlled trial.

1. Problem Statement

Online symptom checkers are established tools for guiding patients to appropriate care levels (self-care vs. professional care). However, they face significant challenges:

• Low Uptake and Adherence: Users often do not follow triage advice, and translation of intention to action is low.
• Health Literacy Barriers: Current designs often use medical jargon, failing to meet the needs of diverse populations and potentially exacerbating health inequities.
• Trust and Transparency: Users frequently distrust tools due to a lack of transparency regarding decision-making processes or concerns about data privacy.
• The AI Gap: While Generative AI (GenAI) offers potential to tailor content and improve plain language communication, there is limited evidence on how explicit AI integration affects user trust, acceptability, and behavioral intentions in a clinical context.

2. Methodology

Study Design:

• Two parallel-group, 5-arm online Randomised Controlled Trials (RCTs).
• Registration: Prospectively registered (ACTRN12625000474459p) and approved by the University of Sydney Ethics Committee.
• Setting: Online survey hosted via Qualtrics.

Participants:

• Sample: 2,110 Australian adults recruited via an online research panel in June 2025.
• Demographics: Median age 49; 51% female, 49% male. Quotas ensured no more than 50% had a university education to reflect the general population.
• Inclusion: Adults (18+), residing in Australia, self-reported good English proficiency.

Intervention (The 5 Arms):
Participants were randomized into one of five groups based on Symptom Acuity (Low vs. Moderate) and Format:

1. Standard: The existing healthdirect symptom checker advice (non-AI, generic advice).
2. AI-Enhanced: Tailored advice, rationale for acuity level, AI disclosure statement, and reassuring tone.
3. AI-Enhanced + Numbered Steps: Added structured, step-by-step self-management instructions.
4. AI-Enhanced + Multimedia: Included a video on fever management and symptom images.
5. AI-Enhanced + Detailed AI Info: Provided explicit, detailed information about how AI was integrated into the tool.

Scenario:
Participants viewed a hypothetical scenario involving fever and vomiting (dehydration risk) and were shown a screenshot of the advice page corresponding to their assigned group.

• Low Acuity: Advice to self-care at home.
• Moderate Acuity: Advice to see a General Practitioner (GP) within 24 hours.

Outcomes:

• Primary: Behavioral intentions (intention to follow self-care advice; intention to see a GP in 24 hours).
• Secondary: Trust (cognitive and emotional), symptom management knowledge (current and changing symptoms), and tool acceptability.
• Timeline: Outcomes measured immediately post-intervention; knowledge measured again at a 2-week follow-up.

Analysis:

• Intention-to-treat analysis.
• Statistical tests: Kruskal-Wallis and ANOVA for group differences; Dunn's test for post-hoc comparisons.
• Sensitivity analyses for attention checks and stratified analyses by demographics (age, education, chronic disease, ChatGPT usage).

3. Key Contributions

• First RCT on GenAI in Symptom Checkers: This is one of the first studies to quantitatively evaluate the impact of GenAI-enhanced communication formats on user behavior and trust in a national health service context.
• Evidence on AI Disclosure: It specifically tests whether explicitly stating that advice is AI-generated negatively impacts trust or acceptability.
• Health Literacy Optimization: It evaluates specific design features (numbered steps, multimedia, tailored rationale) derived from patient and public involvement (PPI) workshops to address health literacy gaps.
• Longitudinal Data: Includes a 2-week follow-up to assess the retention of knowledge gains.

4. Key Results

Behavioral Intentions:

• Low Acuity (Self-Care): Participants in the AI-Enhanced groups reported significantly lower intentions to see a GP within 24 hours compared to the Standard group (Median 3.00 vs. 4.00; adjusted p = 0.003). This suggests AI-enhanced formats may reduce unnecessary primary care visits for manageable symptoms.
• Moderate Acuity (See GP): No significant differences in intentions to see a GP were found between formats.
• Self-Care Intention: No significant differences were found across any group for intentions to follow self-care advice.

Knowledge:

• Immediate Gains: All AI-enhanced formats resulted in significantly higher knowledge scores regarding managing current and changing symptoms compared to the Standard format (adjusted p < 0.001) across both acuity levels.
• Retention: Knowledge gains were not sustained at the 2-week follow-up. In fact, some AI-enhanced groups showed a slightly larger decrease in knowledge retention compared to the Standard group for current symptoms.

Trust and Acceptability:

• No Negative Impact: The explicit use of AI (disclosure statements) did not significantly impact trust or acceptability. Trust and acceptability scores remained high across all formats.
• Stratified Findings: The reduction in GP intention for low-acuity symptoms was most pronounced among participants with a university education, those without chronic conditions, those under 60, and those who had not used ChatGPT recently.

Other Findings:

• No significant differences were found based on gender.
• Sensitivity analysis confirmed results were robust even after removing inattentive participants (9.3% of the sample).

5. Significance and Implications

• Clinical Practice & Policy: The study suggests that integrating GenAI to provide tailored, plain-language advice with clear rationales can improve health literacy and potentially reduce the burden on primary care services for low-acuity conditions.
• Trust in AI: Contrary to fears that AI disclosure might erode trust, the study found that transparent communication about AI use within a reputable national health service (healthdirect) does not negatively affect user trust or acceptability.
• Design Recommendations: To maximize effectiveness, symptom checkers should move beyond generic advice to include:
  • Tailored content specific to the user's symptoms.
  • Clear explanations of why a specific triage level was chosen.
  • Actionable steps (e.g., numbered lists).
• Limitations & Future Directions:
  • The study used static screenshots rather than interactive prototypes, which may limit the generalizability of multimedia findings.
  • Knowledge gains were short-lived, suggesting a need for reinforcement strategies.
  • Future research should test these formats using interactive tools across a wider variety of real-world health contexts.

Conclusion:
AI-enhanced symptom checker formats can effectively improve symptom management knowledge and reduce unnecessary GP visits for self-careable conditions without compromising user trust. Explicit disclosure of AI use is not a barrier to adoption.