Sino-US-DrugQA: A Benchmark for Evaluating Large Language Models in Cross-Jurisdictional Pharmaceutical Regulation

This paper introduces Sino-US-DrugQA, a bilingual benchmark dataset of 11,871 questions derived from US and Chinese pharmaceutical regulations, which reveals that while current large language models perform well on monolingual regulatory queries, they face significant challenges in cross-jurisdictional comparative reasoning, necessitating expert oversight for high-stakes compliance applications.

The Gist

Imagine you are a chef trying to bake the perfect cake, but you have to follow two completely different rulebooks: one written by the US Food and Drug Administration (FDA) and another by China's National Medical Products Administration (NMPA). Both books tell you how to make medicine, but they use different words, different measurements, and sometimes even different rules for the same ingredient.

Now, imagine you hire a super-smart robot chef (an AI) to read both books and tell you the differences. You want to know: Can this robot actually understand the nuances, or will it just guess and get the recipe wrong?

That is exactly what this paper is about. Here is the breakdown in simple terms:

1. The Problem: The "Two-Rulebook" Confusion

In the real world, making medicine is a high-stakes game. If you get the rules wrong, people could get sick. While AI is getting really good at answering medical questions, nobody had really tested if it could handle the tricky job of comparing rules between two different countries (the US and China) at the same time. It's like asking a translator to not just translate a sentence, but to explain why the laws in New York are different from the laws in Beijing.

2. The Solution: A "Training Gym" for AI

The researchers built a massive training gym (a dataset called Sino-US-DrugQA) to test these AI robots.

• The Workout: They created nearly 12,000 quiz questions (multiple-choice) based on real laws from both countries.
• The Levels: Some questions were easy (just asking about US rules), and some were hard (asking, "How is the US rule different from the Chinese rule?").
• The Goal: To see if the AI could act like a seasoned legal expert who knows both rulebooks inside out.

3. The Test: Putting the Robots to Work

They took four of the smartest AI models available (like the latest versions of GPT, Gemini, and others) and asked them to take this quiz without any help (a "zero-shot" test, meaning they couldn't look up answers or get hints).

The Results:

• The Good News: The AI robots did pretty well! They got about 79% to 85% of the answers right. This means they are already very useful for simple tasks, like quickly summarizing a single country's rules for a human expert to check.
• The Bad News: When the questions got harder—requiring the AI to compare the two countries side-by-side—their scores dropped by about 6 to 9 points.

4. The Big Lesson: "Don't Trust the Robot Alone Yet"

Think of the AI like a very fast, very well-read intern.

• If you ask the intern, "What are the US rules for this drug?" they can probably find the answer quickly and accurately.
• But if you ask, "How do the US rules differ from the Chinese rules, and which one is safer?" the intern might get confused or mix up the details.

The paper concludes that while these AI tools are great assistants for drafting documents or screening information, they are not ready to be the final decision-makers when it comes to comparing international laws. Because the stakes are so high (people's health), a human expert must always double-check the AI's work, especially when dealing with cross-border regulations.

In short: The researchers built a giant test to see if AI can handle the complex job of comparing US and Chinese medicine laws. The AI passed the easy parts but stumbled on the hard comparisons, proving that while it's a helpful tool, it still needs a human supervisor to keep everyone safe.

Technical Summary

1. Problem Statement

The pharmaceutical industry faces significant complexity in cross-jurisdictional compliance, specifically requiring the comparative analysis of regulatory frameworks between major authorities like the US FDA (governed by the Code of Federal Regulations, Title 21) and China's NMPA. While Large Language Models (LLMs) are increasingly adopted in healthcare, their ability to perform comparative regulatory reasoning across these distinct legal jurisdictions has not been systematically evaluated. Existing benchmarks lack the specific granularity required to test models on bilingual retrieval and the nuanced deduction needed to compare conflicting or divergent regulatory requirements.

2. Methodology

The study introduces Sino-US-DrugQA, a specialized benchmark designed to fill this gap through the following methodological steps:

• Dataset Construction:
  • Source Material: The dataset is derived from authoritative regulatory texts: NMPA regulations (Chinese) and US CFR Title 21 (English).
  • Scale and Format: It comprises 11,871 multiple-choice question-answer (MCQA) pairs in a bilingual format.
  • Task Types: The questions are categorized into two distinct task types:
    1. Monolingual Retrieval: Testing the model's ability to recall specific regulations within a single jurisdiction.
    2. Cross-Jurisdictional Comparative Reasoning: Testing the model's ability to analyze, compare, and deduce differences or similarities between US and Chinese regulations.
• Experimental Protocol:
  • Models Evaluated: Four state-of-the-art LLMs were tested: GPT-5.2, Gemini-3-flash, Qwen-3-235B, and DeepSeek-V3.2.
  • Evaluation Setting: A standardized zero-shot protocol was used (no few-shot examples provided).
  • Parameters: The temperature was set to 0 to ensure deterministic outputs and reproducibility.

3. Key Contributions

• Novel Benchmark: The creation of Sino-US-DrugQA, the first dedicated bilingual benchmark for evaluating LLMs in the specific domain of cross-jurisdictional pharmaceutical regulation.
• Comprehensive Corpus: A high-quality, large-scale dataset of nearly 12,000 MCQs that bridges the linguistic and regulatory gap between the US and China.
• Open Resources: The authors have released the full dataset and evaluation scripts as open-source resources to facilitate reproducible research in regulatory AI.
• Empirical Baselines: The study establishes the first performance baselines for top-tier LLMs in this high-stakes domain, providing a reference point for future model development.

4. Results

• Overall Performance: In the zero-shot setting, the evaluated models achieved overall accuracy rates ranging from 78.97% to 84.51% across the entire dataset.
• Performance Gap by Task Type:
  • Models performed significantly better on monolingual retrieval tasks.
  • Performance consistently dropped on comparative reasoning tasks, showing a decline of approximately 6 to 9 percentage points compared to monolingual tasks.
• Model Limitations: Even the strongest-performing system struggled with cross-jurisdictional alignment and deduction, indicating that current LLMs have not yet mastered the complex logic required to navigate conflicting regulatory frameworks without error.

5. Significance and Implications

• Practical Utility: The study concludes that current LLMs are viable as drafting and screening assistants for monolingual regulatory queries, where they can efficiently retrieve specific rules.
• Risk Mitigation: Due to the persistent performance gap in comparative reasoning, the authors advocate for a conservative deployment posture. LLMs should not be used for final cross-jurisdictional compliance decisions without mandatory expert human review.
• Future Direction: The benchmark highlights "cross-jurisdictional alignment" and "comparison-specific deduction" as critical areas for future model improvement, guiding the development of more robust regulatory AI systems.

The dataset and evaluation tools are publicly available at https://github.com/DodgeLU/Sino-US-DrugQA.