💬 NLP

PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models

This paper introduces PolicyBench, a large-scale cross-system benchmark for evaluating policy comprehension across memorization, understanding, and application, and proposes PolicyMoE, a specialized Mixture-of-Experts model that demonstrates superior performance on structured reasoning and application-oriented policy tasks.

Original authors: Han Bao, Penghao Zhang, Yue Huang, Zhengqing Yuan, Yanchi Ru, Rui Su, Yujun Zhou, Xiangqi Wang, Kehan Guo, Nitesh V Chawla, Yanfang Ye, Xiangliang Zhang

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Han Bao, Penghao Zhang, Yue Huang, Zhengqing Yuan, Yanchi Ru, Rui Su, Yujun Zhou, Xiangqi Wang, Kehan Guo, Nitesh V Chawla, Yanfang Ye, Xiangliang Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart librarian named LLM (Large Language Model). This librarian has read almost every book, article, and website in existence. They can write poetry, solve math problems, and translate languages instantly.

But there's a catch: Public Policy is a very different kind of book. It's not just about facts; it's about rules, values, who gets what money, and how governments actually work. If this librarian gives you the wrong advice on a tax law or a healthcare rule, real people could lose their homes or healthcare.

The paper you shared, "PolicyLLM," is like a report card and a new training manual for this librarian. Here is the breakdown in simple terms:

1. The Problem: The Librarian is Guessing

The researchers realized that while these AI models are great at general knowledge, they often stumble when it comes to Public Policy.

The Issue: If you ask the AI, "What is the date this law passed?" it might guess. If you ask, "How does this law help a small business in a specific town?" it might make up a plausible-sounding but wrong answer.
The Risk: In the real world, a wrong guess isn't just a bad grade; it's a bad decision that affects society.

2. The Solution Part 1: The "Policy Gym" (PolicyBench)

To fix this, the researchers built a giant gym called PolicyBench. Think of it as a massive obstacle course designed specifically to test how well an AI understands government rules.

The Scope: They didn't just look at one country. They built a course with obstacles from two very different worlds: the United States and China. This tests if the AI can handle different styles of government and language.
The Three Levels of Difficulty (The Bloom's Taxonomy):
- Level 1: The Flashcard Drill (Memorization).
  - Analogy: "What year was the 'Double Reduction' policy signed?"
  - Goal: Can the AI remember the exact facts?
- Level 2: The Detective Work (Understanding).
  - Analogy: "Why did the government write this rule? Who are they trying to help, and what values are they protecting?"
  - Goal: Can the AI understand the spirit and logic behind the rule, not just the words?
- Level 3: The Real-Life Simulation (Application).
  - Analogy: "A small business owner in a specific town has a problem. Based on this policy, what should they do next? How much money can they get?"
  - Goal: Can the AI take the rule and apply it to a messy, real-world situation?

The Result: They found that the AI models are actually better at Level 3 (Application) than Level 1 (Memorization)!

Why? It's like a student who is bad at rote memorization but great at solving puzzles. The AI is good at logical reasoning (solving the puzzle) but sometimes forgets the exact date (the fact). However, they still struggle with abstract concepts and understanding the deep "why" behind policies.

3. The Solution Part 2: The "Specialist Team" (PolicyMoE)

The researchers realized that asking one giant brain to do everything (memorize facts, understand deep concepts, and solve problems) is inefficient. It's like asking a single chef to be a master baker, a grill expert, and a pastry chef all at once.

So, they built PolicyMoE (Mixture of Experts).

The Analogy: Instead of one giant brain, they created a team of three specialists working together:
1. The Archivist: Specializes only in remembering exact dates, names, and numbers.
2. The Philosopher: Specializes in understanding the ideas, values, and "why" behind the laws.
3. The Strategist: Specializes in taking those laws and figuring out what to do in a real-life scenario.
The Router: There is a smart "manager" (a router) that looks at your question. If you ask a date, it sends it to the Archivist. If you ask for advice, it sends it to the Strategist.

The Outcome: This team approach worked much better than the "one-size-fits-all" models. The specialist team made fewer mistakes and gave more reliable advice, especially for complex tasks.

4. The Big Takeaway

The paper teaches us two main things:

We need better tests: We can't just ask AI "What is 2+2?" to see if it's ready for government work. We need specific tests (like PolicyBench) that check if it understands rules, values, and real-world consequences.
Specialization wins: To make AI truly useful for public policy, we shouldn't just make the models "bigger." We should make them smarter by giving them specialized roles, like a team of experts rather than a single generalist.

In a nutshell: The researchers built a rigorous test to see how well AI understands government rules, found that AI is good at solving problems but bad at remembering facts, and then invented a "team of specialists" system to make the AI much more reliable for real-world policy decisions.

1. Problem Statement

While Large Language Models (LLMs) have achieved remarkable success in general NLP tasks, their ability to comprehend and reason about public policy remains under-explored. Public policy analysis requires not only factual knowledge but also contextual reasoning, value-sensitive judgment, and an understanding of complex institutional logic.

The authors identify three critical gaps hindering the deployment of LLMs in policy domains:

Evaluation Challenge: There is no comprehensive, standardized benchmark to systematically assess LLMs' policy comprehension across different cognitive levels (recall vs. reasoning) and governance systems.
Diagnostic Challenge: Aggregate metrics obscure specific strengths and weaknesses, making it unclear which cognitive abilities or policy domains pose the greatest difficulties for models.
Adaptation Challenge: General-purpose LLMs struggle with the distinct, multifaceted demands of policy tasks, and there is a lack of specialized architectures designed to handle these specific requirements.

2. Methodology

A. PolicyBench: A Cross-System Benchmark

To address the evaluation gap, the authors constructed PolicyBench, the first large-scale cross-system benchmark comparing China (CN) and the United States (US).

Data Collection: The dataset comprises 21,000 cases derived from 721 Chinese policies (sourced from the State Council) and 603 US policies (sourced from 12 federal departments), along with supplementary materials (news, expert interviews, commentaries).
Taxonomy (Bloom's Taxonomy): The benchmark categorizes tasks into three cognitive levels:
- Level 1 (Memorization): Factual recall (dates, terminology, organizations).
- Level 2 (Understanding): Conceptual reasoning based on the "3I" framework (Ideas, Interests, Institutions).
- Level 3 (Application): Problem-solving in real-world scenarios (numerical reasoning, decision-making, procedural implementation).
Task Diversity: Includes 10 sub-tasks ranging from multiple-choice and true/false questions to open-ended reasoning.
Quality Control: Distractors are generated using a heterogeneous pool of LLMs to avoid single-model bias. Human experts (Ph.D. candidates in Policy/Law) validated the dataset, achieving high inter-annotator agreement ( $\kappa > 0.80$ ).

B. PolicyMoE: A Domain-Specialized Architecture

To address the adaptation challenge, the authors propose PolicyMoE, a Mixture-of-Experts (MoE) model tailored for policy tasks.

Architecture: Built upon a base model (Qwen2.5-7B-Instruct), it integrates three specialized expert modules, each fine-tuned via LoRA (Low-Rank Adaptation):
1. Memory Expert: Specializes in factual recall and policy language.
2. Understanding Expert: Focuses on interpreting intent, stakeholders, and institutional logic.
3. Application Expert: Excels at scenario-based reasoning and implementation strategies.
Routing Mechanism: A lightweight router (linear layer) analyzes input tokens and dynamically selects the most relevant expert (top-1 selection) to process the query, allowing the model to adapt its "expertise" based on the cognitive level of the task.

3. Key Contributions

PolicyBench: A comprehensive, bilingual (CN/US) benchmark covering diverse policy domains, enabling fine-grained evaluation of LLMs across memorization, understanding, and application.
Diagnostic Insights: Extensive experiments reveal that current LLMs perform well on structured reasoning (application) but struggle with abstract policy concepts and cross-lingual nuances (specifically Chinese policy texts).
PolicyMoE: A novel MoE framework that significantly outperforms general-purpose baselines and standard LoRA fine-tuning by decoupling cognitive tasks into specialized experts.

4. Experimental Results

Benchmark Performance (PolicyBench)

Cognitive Trends: Models show a counter-intuitive trend where performance increases from Level 1 (Memorization) to Level 3 (Application).
- Level 1: Accuracy ranges from ~42% to 62%. Models struggle with precise factual recall due to implicit knowledge storage.
- Level 3: Accuracy ranges from ~68% to 80%. Models excel at structured reasoning (e.g., numerical calculations, scenario-based decisions) likely due to alignment with instruction tuning and RLHF.
Language Bias: Most models perform better on US policies than Chinese policies (e.g., o4-mini: 60.41% CN vs. 65.54% US), attributed to English dominance in pre-training corpora and the higher syntactic density of Chinese policy texts.
Top Performer: DeepSeek-R1 achieved the highest overall accuracy (66.34%) among evaluated models.

PolicyMoE Performance

Significant Gains: Fine-tuning the 7B base model with PolicyMoE resulted in substantial improvements over the base model and standard LoRA baselines.
- Level 1 (Memorization): US tasks saw a massive 51.73% relative improvement (23.35% $\to$ 35.43%).
- Level 3 (Application): US tasks saw a 23.22% relative improvement.
Comparison with Standard LoRA: PolicyMoE outperformed standard LoRA by +2.68% on average, with specific gains of +3.81% on Memorization and +3.85% on Application tasks. This demonstrates that decoupling cognitive tasks into specialized experts mitigates task interference.
Routing Analysis: The router successfully allocated high weights to the "Memory Expert" for Level 1 tasks (>80%) and distributed weights more broadly for Level 2 and 3 tasks, validating the architectural design.

5. Significance and Conclusion

Technical Insight: The study reveals that LLMs' "reasoning" capabilities (Level 3) are often stronger than their "recall" capabilities (Level 1) in policy domains, suggesting that current models rely heavily on pattern matching and instruction-following rather than deep factual grounding.
Architectural Advancement: PolicyMoE demonstrates that domain-adaptive pretraining via specialized MoE architectures is a viable path to improving LLM performance in high-stakes governance tasks without requiring prohibitively large parameter counts.
Future Directions: The authors highlight the need for broader geographic coverage (beyond CN/US), more open-ended task formats, and advanced routing mechanisms that can activate multiple experts simultaneously for complex reasoning.

In summary, PolicyLLM provides a rigorous framework for evaluating and improving LLMs in public policy, offering both a critical diagnostic tool (PolicyBench) and a specialized solution (PolicyMoE) to bridge the gap between general AI and reliable policy analysis.