VietMed-MCQ: A Consistency-Filtered Data Synthesis Framework for Vietnamese Traditional Medicine Evaluation

Imagine you have a super-smart robot librarian (a Large Language Model) that can answer almost any question about modern medicine. It's like a genius who has read every textbook in a Western hospital. But, if you ask it a question about Vietnamese Traditional Medicine (VTM)—which relies on ancient herbs, specific cultural concepts, and local practices—the robot suddenly starts guessing wildly. It's like asking a chef who only knows how to cook Italian pasta to suddenly make a perfect bowl of Phở; they know the basics of cooking, but they lack the specific, cultural "secret sauce."

The main problem is that there aren't enough good "practice tests" for this robot to study. Without a proper test, we don't know if it's actually learning or just making things up.

The Solution: A New "Practice Test" Factory

The authors of this paper built a new factory called VietMed-MCQ to create a massive practice test specifically for Vietnamese Traditional Medicine. Here is how they did it, using a simple analogy:

1. The Recipe Book (RAG Pipeline): Instead of letting the robot guess, they gave it a strict "recipe book" of trusted medical texts. The robot has to look up the answer in the book before writing it down. This is called a Retrieval-Augmented Generation (RAG) pipeline.
2. The Double-Check System: To make sure the robot didn't just hallucinate (make things up), they used a "two-person rule." They had two different AI models look at the same question and try to solve it independently. If both models agreed on the answer, it was likely correct.
   • The Catch: The paper admits this system isn't perfect. It checks if the answer is a "substring" (a piece of text) found in the evidence, which is like checking if a student copied a sentence from the textbook. It's a good start, but it doesn't guarantee the student truly understands the logic behind the sentence.
3. The Human Review: Even with the robots helping, humans are needed. One medical expert and four students reviewed the questions. They gave it a thumbs-up 94.2% of the time, and they mostly agreed with each other (a high "Fleiss' kappa" score), meaning the test is reliable.

The Result: A New Benchmark

They created a test bank of 3,190 multiple-choice questions ranging from easy to very hard. They then put seven different "smart robots" (open-source AI models) through this test to see who would pass.

The Surprising Findings:

• The "Chinese Connection": The robots that were originally trained heavily on Chinese data actually did better than the robots specifically trained on Vietnamese data.
  • The Analogy: Think of it like this: Vietnamese Traditional Medicine shares a lot of roots with Traditional Chinese Medicine. The "Chinese-trained" robots had already studied the "root language" of these medical concepts, so they could translate that knowledge to Vietnamese better than a robot that only knew the Vietnamese language but not the medical history.
• The Struggle: Despite the Chinese advantage, none of the robots were great at complex diagnostic reasoning. They could handle simple facts, but when the question required deep, multi-step thinking (like a real doctor diagnosing a tricky illness), they all stumbled.

The Bottom Line

This paper doesn't claim these robots are ready to replace doctors or treat patients yet. Instead, it's a tool for researchers. They have released the dataset and the code to the public so that other scientists can build better "practice tests" and help these AI models learn the complex, culturally specific world of Vietnamese Traditional Medicine without getting lost.