Evaluation of Large Language Models for Post-Cystectomy Sexual Health Counseling in Women: A Pilot Study

Technical Summary: Evaluation of Large Language Models for Post-Cystectomy Sexual Health Counseling in Women

Problem Statement
Women with muscle-invasive bladder cancer (MIBC) face disproportionately poor survival rates and high rates of sexual dysfunction following radical cystectomy. Despite the clinical significance of these issues, sexual health counseling remains inconsistently delivered, with female patients less likely than males to receive provider-initiated discussions regarding sexual function and psychosocial well-being. While Large Language Models (LLMs) are increasingly utilized for patient education, their reliability in generating guideline-concordant, empathetic, and readable counseling for this specific, sensitive population remains understudied. Existing literature often focuses on clinical accuracy but neglects the "readability gap," where AI-generated content frequently exceeds the health literacy levels of the general population, potentially exacerbating disparities. Furthermore, methodological limitations in prior readability research—specifically the treatment of correlated readability metrics as independent outcomes—risk inflating Type I errors and obscuring true model differences.

Methodology
The study evaluated three widely used LLMs: ChatGPT, Gemini, and Perplexity. The evaluation framework involved the following components:

• Data Generation: The models were prompted with six clinical questions derived from American Urological Association (AUA) and American Society of Clinical Oncology (ASCO) guidelines regarding post-cystectomy care. Prompts were delivered in two formats: long-form (technical/clinician language) and short-form (lay language). To minimize cross-prompt carryover, responses were generated in separate incognito sessions.
• Adherence Assessment: Two independent reviewers scored the 36 generated responses (6 questions × 3 models × 2 reviewers) using a 4-item checklist based on guideline domains: (1) counseling on complications and quality of life, (2) discussion of sexual function-preserving surgical options, (3) supportive care and mental health counseling, and (4) lifestyle modifications. Scores were converted to a proportional adherence metric (0–1).
• Statistical Analysis (Adherence): A linear mixed-effects model was employed to predict guideline adherence as a function of LLM type, prompt format, and reviewer, with the clinical question treated as a random intercept to account for clustering and difficulty variance. Post-hoc pairwise comparisons were conducted using estimated marginal means (EMMs) with Tukey-Kramer correction.
• Readability and Dimensionality Analysis: Five standard readability indices (Flesch Reading Ease, Flesch-Kincaid Grade, Automated Readability Index, SMOG, and Coleman-Liau) were calculated. To address multicollinearity and identify latent structures, Principal Component Analysis (PCA) and hierarchical agglomerative clustering were applied to the readability metrics.

Key Contributions

1. Guideline Adherence Benchmarking: The study provides a comparative assessment of ChatGPT, Gemini, and Perplexity specifically regarding female sexual dysfunction counseling post-cystectomy, a niche often overlooked in broader AI medical evaluations.
2. Prompt Engineering Impact: It demonstrates that prompt framing (lay vs. clinical language) significantly influences the output quality, challenging the assumption that more complex clinical prompts yield better guideline adherence.
3. Methodological Rigor in Readability: The paper introduces a psychometrically grounded analytical pipeline using PCA and hierarchical clustering. This approach validates that standard readability metrics often measure a single underlying construct of linguistic complexity, supporting the use of composite or primary metrics (like Flesch-Kincaid) rather than treating them as independent variables.
4. System-Level Evaluation: The study conceptualizes LLMs as "information systems" transforming clinical prompts into counseling outputs, emphasizing the need to evaluate both the accuracy of the information and the cognitive burden (readability) placed on the patient.

Results

• Model Performance: ChatGPT demonstrated the highest adherence to clinical guidelines (estimated marginal mean [EMM] = 0.769), significantly outperforming Gemini (EMM = 0.499) and Perplexity (EMM = 0.457). No significant difference was found between Gemini and Perplexity.
• Prompt Effect: Short-form, lay-language prompts elicited significantly higher adherence (EMM = 0.667) compared to long-form, clinician-worded prompts (EMM = 0.483).
• Readability: All models produced content exceeding recommended health literacy levels. Perplexity generated the most complex text (approximate Flesch-Kincaid Grade Level ~16), followed by ChatGPT, with Gemini producing the lowest grade-level content (approx. Grade 11). However, even the "simpler" outputs remained above the recommended sixth-grade reading level.
• Dimensionality: PCA revealed that a single dominant component (PC1) accounted for 76.7% of the variance across the five readability indices, confirming that these metrics largely reflect a shared underlying construct of textual difficulty.

Significance and Claims
The paper claims that LLMs function as variable medical information systems whose outputs are heavily dependent on both the specific model architecture and the structure of the input prompt. The findings suggest that while ChatGPT currently offers the most guideline-concordant information for this specific clinical scenario, high linguistic complexity remains a universal barrier to patient comprehension across all tested models.

The authors emphasize that the study characterizes the current state of AI in this domain as one where "high linguistic complexity was seen across models, highlighting a barrier to patient comprehension." They conclude that while LLMs hold potential to supplement counseling, their deployment in sensitive contexts like post-cystectomy sexual health requires cautious, clinician-supervised oversight. The study does not claim that LLMs are ready for autonomous clinical use but rather that they serve as tools whose reliability varies significantly based on model selection and prompt design. The research underscores the necessity of evaluating both the content (guideline adherence) and the form (readability) of AI-generated health information to prevent the amplification of existing health disparities.