💻 computer science

Green Shielding: A User-Centric Approach Towards Trustworthy AI

This paper introduces "Green Shielding," a user-centric framework utilizing the CUE criteria and the HealthCareMagic-Diagnosis benchmark to demonstrate how routine, non-adversarial variations in user prompts systematically shift large language model outputs in medical diagnosis, revealing critical trade-offs between output plausibility and safety-critical coverage to guide trustworthy AI deployment.

Original authors: Aaron J. Li, Nicolas Sanchez, Hao Huang, Ruijiang Dong, Jaskaran Bains, Katrin Jaradeh, Zhen Xiang, Bo Li, Feng Liu, Aaron Kornblith, Bin Yu

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aaron J. Li, Nicolas Sanchez, Hao Huang, Ruijiang Dong, Jaskaran Bains, Katrin Jaradeh, Zhen Xiang, Bo Li, Feng Liu, Aaron Kornblith, Bin Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are asking a very smart, well-read but slightly nervous librarian for help finding a book. If you ask, "I have a headache," the librarian might panic and suggest every possible cause from a hangover to a brain tumor. But if you ask, "I have a headache and I've been drinking too much coffee," the librarian might calmly suggest a caffeine withdrawal.

This paper, titled "Green Shielding," argues that current AI safety tests are like hiring a "red team" to break into the library and steal books (finding extreme, malicious failures). However, they aren't testing what happens when regular people ask questions in slightly different, everyday ways. The authors propose a new approach called Green Shielding: studying how harmless, routine changes in how users ask questions change the AI's answers, especially in high-stakes fields like medicine.

Here is a breakdown of their findings using simple analogies:

1. The Problem: The "Fickle Librarian"

The authors found that Large Language Models (LLMs) are surprisingly sensitive to how you phrase a question, even if the medical facts stay the same.

The Analogy: Imagine a doctor who gives you a different diagnosis depending on whether you sound worried, whether you list your symptoms in a messy paragraph or a neat list, or whether you mention a specific guess you have in mind.
The Finding: In their tests, simply changing the tone (adding urgency), the format (making it a multiple-choice question), or the content (removing a lab result) caused the AI's answers to flip from "correct" to "incorrect" or vice versa. The AI isn't just unstable; it's predictably unstable based on these small user choices.

2. The Solution: The "Green Shield"

Instead of just trying to stop the AI from being evil (Red Teaming), the authors want to build a "Green Shield"—a user manual based on evidence. They created a framework called CUE to test this:

Context: Use real patient stories, not fake exam questions.
Utility: Measure what actually matters (did the AI catch the dangerous diseases?), not just if it got the "right" single answer.
Elicitation: Test how the AI reacts when you tweak the input in realistic ways.

3. The Experiment: The "Medical Translator"

To test this, the researchers built a new dataset called HCM-Dx.

The Setup: They took thousands of real, messy questions from patients (who often sound scared, ramble, or miss details) and fed them to top-tier AI models.
The Twist: They also built a "Translator" (Prompt Neutralization). This tool took the messy, emotional patient questions and rewrote them into clean, objective, third-person medical notes (e.g., changing "I'm so scared, my jaw hurts!" to "Patient reports right-sided jaw pain for 2 days").

4. The Big Discovery: The "Precision vs. Safety" Trade-off

This is the most important part of the paper. When they compared the AI's answers to the messy questions vs. the clean, neutralized questions, they found a Pareto trade-off (a situation where you can't improve one thing without hurting another).

The Messy (Real) Input: The AI acted like a nervous student. It listed many possibilities (high coverage). It was good at catching the scary, life-threatening diseases, but it also listed a lot of unlikely things, making the answer long and confusing.
The Clean (Neutralized) Input: The AI acted like a calm, experienced doctor. It gave a short, concise list of the most likely diagnoses (high plausibility). However, in its effort to be concise and "clinician-like," it started missing the rare but deadly safety-critical diseases.

The Metaphor:
Think of the AI as a security guard at a gate.

When the guard is stressed by a chaotic crowd (messy prompts), they check everyone and let almost no one slip through, but they also stop a lot of innocent people (low precision, high safety coverage).
When the guard is given a calm, organized list of names (neutralized prompts), they let the right people through quickly and ignore the noise. But because they are so focused on efficiency, they might accidentally let a dangerous person slip through because they didn't check the "long shot" possibilities.

5. What This Means for You

The paper concludes that there is no single "best" way to ask an AI a medical question.

If you want the AI to be concise and sound like a doctor, you should phrase your question neutrally. But you risk missing some rare, dangerous conditions.
If you want the AI to be thorough and catch every possible danger, you might need to provide more context or express urgency, but the answer will be longer and include more "noise."

The Bottom Line:
"Green Shielding" doesn't tell you which AI answer is perfect. Instead, it provides a map showing that how you ask the question changes the AI's behavior. It proves that small, everyday choices in how we talk to AI can systematically shift whether the AI is "safe" (catches everything) or "precise" (gives a short answer), and we need to understand these trade-offs before trusting AI in real life.

1. Problem Statement

Current AI safety research predominantly focuses on AI Safety I (Red-Teaming), which probes models under adversarial conditions to identify catastrophic failures (e.g., jailbreaking, prompt injection). However, this approach overlooks AI Safety II: the reliability and risks arising from benign, non-adversarial variations in routine user inputs.

In high-stakes domains like healthcare, users (patients) provide queries that vary naturally in tone, format, specificity, and emotional framing. Existing benchmarks (e.g., MedQA) are often exam-style, single-answer, and information-complete, failing to capture the ambiguity and noise of real-world patient interactions. Consequently, there is a lack of evidence-backed guidance on how these routine input variations shift model behavior, potentially leading to unreliable or unsafe outputs in real deployments.

2. Methodology: The Green Shielding Framework

The authors propose Green Shielding, a user-centric research agenda grounded in the PCS (Predictability, Computability, Stability) framework. To operationalize this, they introduce the CUE criteria:

Context: Tasks representing authentic real-world use.
Utility: Reference standards and metrics measuring task-relevant utility.
Elicitation: Perturbation regimes reflecting realistic variations in user inputs.

A. Benchmark Creation: HealthCareMagic-Diagnosis (HCM-Dx)

To address the "Context" gap, the authors curated HCM-Dx, a novel benchmark derived from the HealthCareMagic-100K dataset.

Source: 2,697 real patient-authored queries filtered for explicit diagnostic requests.
Characteristics: Unlike exam questions, these are open-ended, often vague, and contain emotional cues or irrelevant details.

B. Structured Reference Standards

To address the "Utility" gap, the authors moved beyond single-answer evaluation to a set-valued reference structure for differential diagnosis:

Plausible Set ( $P$ ): All medically plausible diagnoses.
Highly Likely Set ( $H$ ): Top working hypotheses.
Safety-Critical Set ( $S$ ): Severe conditions that must be ruled out immediately (cannot-miss diagnoses).

Construction: An ensemble of frontier LLMs (GPT-5, Gemini-3, Claude-4) generates these sets based on structured clinical representations extracted from raw inputs. Consensus voting is used to establish ground truth.

C. Evaluation Metrics

The framework employs both structural and semantic metrics:

Structural: Plausibility (fraction of output in $P$ ), H-coverage and S-coverage (recall of critical sets), and Breadth (number of diagnoses).
Semantic: Evidence grounding (is the diagnosis supported by input?), Indirect inference (does the model hallucinate patient-specific facts?), and Epistemic uncertainty (does the model admit uncertainty?).

D. Prompt Neutralization

To address the "Elicitation" gap, the authors developed a Prompt Neutralization module. This system rewrites raw, noisy patient queries into standardized, third-person clinical descriptions while preserving core medical facts but removing user-level factors (e.g., urgency, anxiety, specific guesses, unstructured formatting). This allows for controlled measurement of how routine input variations affect model performance.

3. Key Contributions

Green Shielding Agenda: A conceptual shift from adversarial red-teaming to studying benign input variations to generate actionable deployment guidance.
HCM-Dx Benchmark: A scalable, patient-authored dataset for open-ended diagnostic reasoning, bridging the gap between exam-style QA and real-world clinical queries.
CUE-Guided Evaluation: A rigorous framework using set-valued references and fine-grained metrics to evaluate differential diagnosis lists rather than single answers.
Prompt Neutralization Tool: A mechanism to isolate the impact of user-level factors (tone, format, content) on model behavior.

4. Key Results

The study evaluated multiple frontier LLMs (GPT-4.1/5-mini, Gemini-3, Claude-4, DeepSeek) on HCM-Dx.

A. Sensitivity to Input Variation

Pilot Study: On standard benchmarks (MedQA, MedXpertQA), minor changes in content (removing symptoms), format (hiding options), or tone (adding urgency) caused significant fluctuations in accuracy (success rates of 10–30%).
HCM-Dx Findings: Raw patient inputs lead to models generating moderately sized differential lists with high plausibility (most guesses are medically valid) but low coverage of highly likely and safety-critical conditions.

B. The Precision-Coverage Tradeoff (Pareto-like)

The most significant finding is a systematic tradeoff induced by Prompt Neutralization:

Effect of Neutralization: When user-level noise (anxiety, unstructured format) is removed, models produce more concise lists with higher plausibility (closer to clinician style).
The Cost: This conciseness comes at the expense of recall. Neutralized prompts significantly reduce coverage of the Highly Likely ( $H$ ) and Safety-Critical ( $S$ ) sets.
Interpretation: "Raw" prompts (with anxiety/urgency) trigger models to be more exhaustive (higher coverage, lower precision), while "Neutral" prompts trigger models to be more selective (higher precision, lower coverage).

C. Comparison with Clinicians

Clinicians: Produce narrow, prioritized lists (low breadth) and rely heavily on indirect inference (making assumptions based on context).
LLMs: Tend toward exhaustive enumeration to hedge uncertainty.
Convergence: Prompt neutralization shifts LLM behavior closer to the clinician style (more selective), but the models still struggle to reliably surface safety-critical conditions without explicit prompting.

5. Significance and Implications

Beyond Single-Score Metrics: The paper demonstrates that aggregate accuracy metrics obscure critical tradeoffs. A model might be "accurate" on a single answer but fail to flag a life-threatening condition in a differential list.
Actionable Guidance: Green Shielding provides a methodology to define "operating points." For example, in a triage setting, one might prefer "Raw" prompts to maximize safety-critical coverage, whereas in a specialist consultation, "Neutral" prompts might be preferred for precision.
Regulatory Relevance: The framework aligns with emerging regulations (e.g., EU AI Act, NIST AI RMF) by providing evidence-backed, deployment-relevant behavioral assessments rather than just theoretical safety checks.
Generalizability: While instantiated in medicine, the approach applies to any decision-support domain (law, education) where user input variation shapes downstream reasoning and actions.

Conclusion

"Green Shielding" argues that for AI to be trustworthy in high-stakes environments, we must understand how benign, routine variations in user interaction alter model behavior. By characterizing the tradeoffs between precision and coverage through the CUE framework, the authors provide a path toward developing interaction strategies and deployment guidelines that are robust, context-aware, and safe.