Prompting is All You Need: How to Make LLMs More Helpful for Clinical Decision Support

This study demonstrates that structured prompting (specifically the CARDS framework) significantly enhances the safety, guideline adherence, and clinical reasoning of both closed-source and open-source large language models for acute stroke thrombolysis decision support, though performance gains vary by model architecture.

The Gist

Imagine you have a team of incredibly smart, super-fast librarians (these are the Large Language Models, or LLMs). These librarians have read almost every book in the world, including medical textbooks. You want to ask them a very serious question: "Should we give this specific stroke patient a powerful clot-busting drug?"

If you just ask them casually, like, "Hey, should we give the drug?", they might give you a quick answer. Sometimes it's right, but sometimes they might miss a tiny detail that makes the answer dangerous.

This paper is like a report card on how to talk to these librarians to get the best, safest, and most helpful answers.

The Big Experiment: "Just Ask" vs. "The 5-Step Checklist"

The researchers tested six different "librarians" (three from big tech companies like OpenAI, and three open-source ones anyone can use). They gave them three fake stroke patient stories and asked the same question in two different ways:

1. The "Casual Chat" (Simple Prompt): Just the story and the question: "Should we give the drug?"
2. The "Structured Checklist" (CARDS Prompt): A specific, five-step instruction that forces the librarian to:
   • Context: Read the story carefully.
   • Aims: Know exactly what the goal is.
   • Relevant details: Pull out the specific numbers and dates.
   • Design: Check the rules (guidelines) and look for red flags (contraindications).
   • Source: Explain why they made the decision and talk about the risks vs. benefits.

Think of the Simple Prompt like asking a chef, "Make me dinner." They might make something good, or they might forget the salt, or use an ingredient you're allergic to.
Think of the Structured Prompt like giving the chef a recipe card: "First, check the allergies. Second, chop the veggies. Third, season with salt. Fourth, cook for 10 minutes. Finally, explain why this dish is healthy."

What Happened?

The results were like a magic trick for some of the librarians, but a bit more modest for others.

1. The "Super-Reasoners" (GPT-4o, o3, GPT-5.2 Thinking, and R1-1776)
These models were already pretty good, but when you gave them the 5-Step Checklist, they became perfect.

• Before: They followed the rules 83% of the time. Sometimes they gave a dangerous "unsafe" answer.
• After: They followed the rules 100% of the time. They stopped giving dangerous answers completely. They started explaining their reasoning clearly, like a doctor talking to a patient.
• The Metaphor: It's like taking a brilliant student who sometimes daydreams and giving them a strict study guide. Suddenly, they ace the test every single time.

2. The "Hard-Working but Flawed" (Llama models)
These open-source models tried their best, but the checklist didn't fix everything.

• Before: They followed rules 66% of the time and gave dangerous advice 33% of the time.
• After: They got better at spotting risks and explaining things, but they still gave dangerous advice 33% of the time. They followed the rules only 66% of the time.
• The Metaphor: Imagine a very hardworking intern who is great at organizing files but keeps forgetting to lock the front door. Giving them a checklist helps them organize better, but they still forget to lock the door. They need more training (or a different kind of brain) to be fully safe.

The Main Takeaway

"Prompting is All You Need" (mostly).

The paper concludes that how you ask the question matters more than just having a smart computer.

• If you use a structured, step-by-step prompt (like the CARDS method), you can turn a smart AI into a highly reliable medical assistant for stroke decisions.
• Some AI models (the "Reasoning" ones) are so good that with the right prompt, they are as safe as a human expert.
• Other AI models are still a bit risky, even with a good prompt.

The Golden Rule for Doctors

Even with the best prompts and the smartest AI, humans must still be in the loop.

Think of the AI as a co-pilot in a plane. The co-pilot can read the instruments, check the weather, and suggest a route. But the human pilot (the doctor) must keep their hands on the controls and make the final decision. The paper warns us: Don't let the AI fly the plane alone, no matter how good the instructions are.

In short: If you want an AI to help with serious medical decisions, don't just chat with it. Give it a strict, step-by-step checklist. It will make the AI smarter, safer, and much more helpful.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) show promise as clinical decision support (CDS) tools, particularly for parsing free-text clinical data to identify contraindications for time-sensitive interventions like thrombolysis (tPA) in acute ischemic stroke. However, early evaluations indicate that LLMs queried with simple, unstructured prompts often exhibit:

• Variable accuracy in adhering to clinical guidelines.
• Unsafe recommendations (e.g., suggesting thrombolysis when contraindicated).
• Lack of explanatory depth (failure to cite guidelines or explain risk-benefit trade-offs).

While prompt engineering is known to improve LLM behavior, there is a lack of formal exploration regarding best practices for realistic, high-stakes neurology settings, specifically comparing the efficacy of structured versus simple prompting across both proprietary (closed-source) and open-source model families.

2. Methodology

The study employed a controlled evaluation of six contemporary LLMs using synthetic clinical vignettes.

• Models Evaluated:
  • Closed-Source (Proprietary): OpenAI GPT-4o, OpenAI o3, and OpenAI GPT-5.2 Thinking.
  • Open-Source: Meta Llama-4-Scout-17B-16E-Instruct, Llama-3.3-70B-Instruct-Turbo, and Perplexity R1-1776 (a fine-tuned DeepSeek-R1 variant with censorship removed).
• Data: Three novel, synthetic ischemic stroke vignettes constructed by the authors (not based on real patients).
• Prompting Strategies:
  1. Simple Prompt: The vignette followed by a single question: "Should this patient be offered thrombolytics?"
  2. Structured Prompt (CARDS Framework): A five-step rubric guiding the model through:
     • Context: Extracting critical information.
     • Aims: Analyzing timing (onset to treatment).
     • Relevant details: Checking for contraindications.
     • Design: Explaining the decision process.
     • Source: Discussing risks and benefits.
• Evaluation Metrics: A board-certified stroke neurologist manually rated outputs across seven domains:
  1. Guideline adherence (Yes/No).
  2. Safety of recommendations (Yes/No).
  3. Recognition of key risks (Yes/No).
  4. Accuracy of specific guideline grading (Yes/No).
  5. Inclusion of conversational explanation (Yes/No).
  6. Clarity (1–5 scale).
  7. Overall helpfulness (1–5 scale).
• Execution: Queries were sent via APIs (AvoMD, Together AI, Perplexity Labs) with default parameters.

3. Key Contributions

• Empirical Validation of CARDS Framework: The study demonstrates that a specific, five-step structured prompt (CARDS) significantly outperforms simple questioning for thrombolytic decision support.
• Comparative Analysis of Model Architectures: It provides a granular comparison between proprietary and open-source models, highlighting that "reasoning" models (both proprietary and open-source) respond differently to prompting than standard instruction-tuned models.
• Safety and Adherence Benchmarking: The paper establishes baseline safety metrics for current frontier models in acute stroke scenarios and quantifies how much prompt engineering can mitigate safety risks.

4. Key Results

The impact of structured prompting varied significantly by model family:

A. Closed-Source Models (GPT-4o, o3, GPT-5.2 Thinking)

• GPT-4o & o3: Structured prompts eliminated unsafe recommendations (dropping from 16.7% to 0%) and achieved 100% guideline adherence. Most notably, guideline grading accuracy jumped from 0% to 100%, and conversational explanations appeared in 100% of outputs (vs. 0% previously).
• GPT-5.2 Thinking: Already exhibited perfect safety and risk recognition under simple prompting. Structured prompts improved guideline adherence from 66.7% to 100% and grading accuracy from 0% to 100%.

B. Open-Source Reasoning Model (R1-1776)

• Achieved 100% adherence and 0% unsafe recommendations even with simple prompting.
• Structured prompts drove guideline grading accuracy and conversational explanations from 0% to 100%.
• Helpfulness scores increased from 3.7 to 5.0 ("Very Good").
• Significance: This model demonstrated that open-source reasoning models can match proprietary performance when properly prompted.

C. Non-Reasoning Open-Source Models (Llama-4-Scout, Llama-3.3-70B)

• Showed modest gains compared to reasoning models.
• Risk recognition improved (83.3% to 100%) and guideline grading improved (0% to 66.7%).
• Critical Limitation: Guideline adherence remained stagnant at 66.7%, and unsafe recommendations persisted at 33.3% even with structured prompts.
• Interpretation: These models likely lack the inherent alignment or training data depth to fully overcome safety gaps via prompting alone.

Overall Trends

• Structured prompting yielded the largest absolute gains in guideline grading accuracy (0% to 100% for top models) and conversational reasoning.
• Closed-source models and the open-source reasoning model (R1-1776) were the only ones to achieve perfect safety and adherence with structured prompts.

5. Significance and Implications

• Clinical Deployment: For immediate clinical use, structured prompting is non-negotiable. It transforms LLMs from "black box" guessers into systematic reasoning engines that explicitly check contraindications and cite guidelines.
• Model Selection:
  • Proprietary models (GPT-4o, o3, GPT-5.2) and Open-source reasoning models (R1-1776) are currently the most viable candidates for high-stakes CDS when paired with structured prompts.
  • Standard open-source models (Llama family) may require Supervised Fine-Tuning (SFT) on stroke-specific corpora or Retrieval-Augmented Generation (RAG) before they are safe for clinical deployment, as prompting alone did not fully resolve their safety deficits.
• Human-in-the-Loop: Despite the improvements, the authors emphasize that vigilant human oversight remains essential. No model achieved 100% perfection across all metrics in all categories without structured prompting, and even with prompting, human verification is critical for life-or-death decisions.
• Future Research: The study calls for larger vignette sets, multiple expert raters, and longitudinal tracking of model versions to see if intrinsic safety gaps are narrowing naturally.

Conclusion: The paper concludes that while "Prompting is All You Need" is a strong driver for performance, the type of model matters. Structured prompting unlocks the latent clinical knowledge in reasoning models (both proprietary and open-source), but for non-reasoning open-source models, it is a necessary but insufficient condition for safe clinical deployment.