Context-Dependent FHIR Serialisation Strategies for Clinical LLM Deployment: A Multi-Layer Benchmark on UK Core Data
This study introduces FHIRBench-UK to demonstrate that the optimal FHIR-to-text serialisation format for clinical LLMs is context-dependent rather than universal, revealing that task-aware routing significantly improves performance across diverse models and tasks while highlighting a critical disconnect between token-level metrics and clinical quality.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
Imagine you are trying to teach a super-smart robot how to understand a patient's medical history. The doctor has all the information stored in a giant, complex digital filing cabinet called FHIR (Fast Healthcare Interoperability Resources). Think of FHIR like a massive, nested set of Russian dolls or a deeply organized spreadsheet where every piece of data—like a medication name, a blood pressure reading, or a doctor's note—is tucked inside specific folders and sub-folders. This is how hospitals store data so computers can talk to each other.
However, the robot you want to hire is a Large Language Model (LLM). This is a type of AI that is incredibly good at reading and writing human language, but it gets confused by raw computer code or nested folders. It can't "read" the filing cabinet directly; it needs the information translated into a story, a list, or a letter that a human would read. The big question researchers have been asking is: How do you translate this messy digital filing cabinet into text for the robot? Does it matter if you just dump the whole computer file as-is, or if you rewrite it as a doctor's letter, or if you turn it into a neat table?
This is the heart of the new study by Jacqueline Chong from The Hong Kong Polytechnic University. The team wanted to find out if the way we "translate" patient data changes how well the AI performs its job. They weren't just guessing; they ran a massive experiment with 100 different patient records, five different AI models, and six different ways of translating the data. They tested the AI on three real-world tasks: answering specific questions (like "What is the patient's NHS number?"), figuring out if drugs might clash (clinical reasoning), and writing a summary letter for a new doctor.
Here is the twist they discovered: There is no single "best" way to translate the data. It depends entirely on what the AI is being asked to do.
If the AI needs to find a specific fact, like a drug code or a date, the best method is to leave the data in its original, messy computer format (raw_json). It's like giving the robot the exact spreadsheet so it can find the specific cell without any translation errors. But, if the AI needs to think about the data—like spotting that a patient is taking two drugs that shouldn't be mixed together—leaving it in raw computer code is actually the worst choice! In those cases, rewriting the data into a structured, organized format (like a clear medical letter or a table) helps the AI understand the relationships between the facts much better.
The researchers found that for 58% of the situations they tested, the default method (just dumping the raw computer file) was actually suboptimal. By simply changing how they presented the data to the AI, they could make the AI significantly smarter at specific tasks without needing to upgrade the AI itself. For example, when asking the AI to write a summary, using a structured format with headings (like a well-organized report) made the AI perform just as well as the raw data but used 88% less computer memory and cost.
One of the most surprising findings was about the AI models themselves. The study showed that the "smarter" and more expensive AI models (like Claude Sonnet 4.5) were very robust; they could handle messy data formats almost as well as clean ones. However, the cheaper, mid-tier models (like Llama 3.3) were much more sensitive. For these budget-friendly models, the way you present the data made a huge difference. Using the wrong format could make a cheap model perform poorly, but using the right format could boost its performance to be almost as good as the expensive ones. This suggests that hospitals don't necessarily need to buy the most expensive AI; they just need to be smarter about how they feed it data.
The team also tested their results with "noisy" data—simulating real-world problems like missing information, duplicate entries, or messy handwriting. The results held up: the best translation method for a specific task remained the same even when the data was imperfect.
Finally, the study confirmed a strange phenomenon: the AI models that looked the best on standard computer tests (which just check if words match) were actually the worst at producing safe, useful medical advice. Conversely, the models that seemed "worse" on those simple tests were often the ones producing the most clinically helpful responses. This warns us that we can't just rely on simple math scores to judge medical AI; we have to look at how well it actually helps a doctor.
In short, this paper proves that how you prepare data for an AI is just as important as the AI itself. It's not a "one-size-fits-all" solution. If you want an AI to be a great fact-finder, give it the raw data. If you want it to be a great thinker or writer, give it a clean, organized story. By matching the translation style to the task, we can make clinical AI safer, cheaper, and more effective for everyone.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.