A System 1 Decision Model for Medical Multiple-Choice Questions Built on Frozen Language Models
This paper presents a fast, cost-effective System 1 decision model for medical multiple-choice questions that achieves up to 46% accuracy on MedMCQA by training only lightweight projection heads on top of frozen language models, establishing a strong baseline while acknowledging it remains below the performance of more complex systems like Med-PaLM 2.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast landscape of artificial intelligence, researchers often distinguish between two ways a machine might think. One is a slow, deliberate process where the system reasons step-by-step, weighing evidence and constructing an argument before speaking. The other is a fast, automatic reaction, where the system makes an immediate judgment based on a gut feeling or a quick pattern match. This second type of thinking, often called "System 1," is what allows humans to recognize a face in a crowd or catch a falling cup without conscious calculation. For computers, achieving this kind of speed usually means simplifying the task: instead of generating a long explanation, the machine simply looks at a question and a list of possible answers, then picks the one that feels most similar to the question. The challenge for scientists is to see if this fast, intuitive approach can handle complex, high-stakes knowledge, such as the vast and intricate field of medicine, where a wrong guess can have serious consequences.
A researcher set out to test the limits of this fast-thinking approach on medical multiple-choice questions. They built a system designed to answer exam questions in a single pass, without generating any text or reasoning steps. The core of their experiment was a clever division of labor. They took powerful, pre-existing language models—large computer programs trained on massive amounts of text—and froze them, meaning they did not change the internal knowledge of these models at all. Instead, they treated these frozen models as static libraries of knowledge, using them only to turn words into mathematical points in space. On top of this frozen foundation, they trained two tiny, specialized components. One component learned to translate a medical question into a point, and the other learned to translate each possible answer choice into a point. The system's job was then simple: find the answer point that sat closest to the question point in this mathematical space. If the answer was close, the system picked it; if not, it moved on.
They tested this method on a dataset of over 190,000 questions drawn from Indian medical entrance exams, covering twenty-one different medical subjects. They started with a basic setup where the system simply looked at the last word of the question and the answer to make its decision. In this initial state, the system performed only slightly better than random guessing, getting about one question right out of every three. This result was expected, as the raw output of a language model is not naturally designed to summarize a whole sentence into a single point. However, they discovered that the way they asked the model to process the text made a massive difference. By wrapping the questions and answers in a simple prompt that asked the model to summarize the meaning in just one word, they forced the model to compress the information more effectively. This small change, combined with a slight adjustment to how the tiny components were trained, caused the system's performance to jump significantly.
With these improvements, the system began to understand the questions much better. When tested on a set of nearly 4,000 questions it had never seen before, the best version of the system, built on a larger language model, answered 46.0% of the questions correctly. This was a substantial improvement over the initial 33% and far above the 25% accuracy one would expect from pure chance. They confirmed that the system was actually reading the questions and not just memorizing which answers appeared most often in the training data. When they scrambled the questions so that the answers belonged to different questions entirely, the system's performance dropped back down to the level of random guessing, proving that it was indeed connecting the specific question to the specific answer.
Despite these gains, the study also highlighted the boundaries of this fast-thinking approach. The system struggled significantly with questions that asked what was not true, or those containing words like "except" or "least." In these cases, the system's accuracy dropped by a wide margin, suggesting that its method of measuring similarity works best for positive statements and falters when the logic requires negation. Furthermore, while the system was fast and efficient, taking only minutes to train and seconds to run, it still fell far short of the most advanced medical AI systems available today. Those top-tier systems, which use slow, deliberate reasoning and generate full explanations, achieve accuracy rates above 70%. They concluded that their fast, single-pass model serves as a valuable, low-cost baseline for understanding how machines can make quick decisions, but it is not yet a competitor to the most sophisticated medical diagnostic tools.
The work also revealed that the size of the underlying language model mattered, but only after the system was taught how to process the text correctly. A smaller model and a medium-sized model performed almost identically when using the basic method, but the largest model pulled ahead once the "one-word summary" technique was applied. This suggests that larger models hold more potential for this kind of task, provided they are guided to extract the right kind of information. They noted that their method is incredibly cheap to run, requiring only a fraction of the computing power needed by the largest generative models. They view their work not as a final solution, but as a clear demonstration of how far a frozen, fast-thinking system can go on its own, and where the next steps—such as allowing the main model to learn slightly or adding a step to handle negative questions—might lead. The study stands as a concrete measurement of the trade-off between speed and depth in medical artificial intelligence.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.