CRAFT: LLM-Based Iterative Refinement for Temporal Reasoning over Clinical Narratives
The paper proposes CRAFT, an LLM-based framework that iteratively refines stage-wise symptom timelines by pairing a generator with a constraint-based verifier, demonstrating improved temporal ordering accuracy on a new benchmark of vaccine adverse-event narratives.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to solve a mystery where the clues are scattered across a messy diary. The diary doesn't have dates or timestamps; instead, it just says things like "I felt sick," "then my head hurt," and "later I threw up." Your job is to figure out the exact order in which these things happened. This is the daily challenge for doctors and scientists trying to understand how diseases progress or how vaccines affect the body. They need to know not just what happened, but when it happened relative to other events. In the world of computer science, this is called "temporal reasoning." It's like trying to reconstruct a movie scene from a pile of loose, unnumbered film strips. If you get the order wrong, you might think the fever caused the rash, when actually the rash came first and caused the fever. Getting this timeline right is crucial for keeping people safe and understanding how illnesses move through the body, but it's incredibly hard because human stories are often vague and full of "before," "after," and "meanwhile" without any hard numbers.
This is where a new tool called CRAFT comes in. Think of CRAFT as a super-smart detective team working on a case. Instead of just one detective guessing the timeline, CRAFT uses two AI "agents" that work together in a loop. The first agent, the Generator, is like a creative writer who looks at a patient's story and tries to draft a timeline of symptoms. The second agent, the Verifier, is like a strict editor who checks that draft. The editor doesn't just say "good job" or "bad job"; they give specific, targeted feedback, like, "You put the headache and the vomiting in the same time bucket, but the story says they happened on different days." The writer then rewrites the timeline based on that feedback, and the editor checks again. They keep doing this—draft, critique, rewrite—until the timeline is perfect or they run out of attempts.
The researchers tested this system on a massive new collection of stories called MedTempo, which contains 5,347 real reports of people having reactions to COVID-19 vaccines. These reports are tricky because they are single stories without clear dates, just like the messy diary in our analogy. The team found that this "draft-and-edit" loop worked much better than letting the AI guess the timeline just once. In fact, the system improved its accuracy significantly as it went through more rounds of feedback. They tested this on four different AI "brains" (ranging from very powerful ones to slightly smaller ones), and in every case, the CRAFT method helped the AI get the timeline right more often.
However, the paper also points out that not all AI brains learn the same way from this feedback. The most powerful AI models (like the ones from Claude and GPT) were able to use the editor's notes to make big improvements, fixing their mistakes over several rounds. But some of the smaller or less capable models hit a wall; they either got the answer right on the very first try and didn't need more help, or they got confused by the feedback and made their answers worse. The researchers also discovered that the difficulty of the stories varied depending on which vaccine the patient received; stories about one specific vaccine were consistently harder for the AI to untangle than others, suggesting that the way people write about different vaccines might be subtly different.
Ultimately, the paper shows that giving AI a chance to "think again" with a helpful critic is a winning strategy for untangling complex medical stories. It's not just about having a smart AI; it's about giving that AI a structured way to check its own work. The authors suggest that while this method is a big step forward, there is still work to be done to make sure the AI knows when to stop editing and when to trust its first instinct, especially for different types of medical reports. This approach could eventually help doctors and safety monitors spot dangerous patterns in patient stories much faster and more accurately than before.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.