TaxPraBen: A Scalable Benchmark for Structured Evaluation of LLMs in Chinese Real-World Tax Practice

1. Problem Statement

While Large Language Models (LLMs) have achieved significant success in general domains, they exhibit critical gaps in highly specialized, knowledge-intensive, and legally regulated fields like Chinese taxation. Existing evaluation benchmarks suffer from several limitations:

• Task Isolation: Most benchmarks (e.g., TaxBen, FinBen) focus on isolated NLP tasks (classification, generation, or simple reasoning) rather than holistic, real-world workflows.
• Lack of Structured Output: They often neglect the need for structured, hybrid outputs that combine semantic reasoning with precise numerical calculations, which are essential in tax practice.
• Data Scarcity & Quality: There is a lack of high-quality, annotated tax datasets specific to China's unique regulatory environment. Existing data often relies on public datasets that may not reflect real-world scenarios or lack open-source accessibility.
• Evaluation Flaws: Current metrics often overstate model capabilities by averaging performance on simple tasks, failing to capture the complexity of integrating legal interpretation with quantitative tax planning.

2. Methodology

The authors introduce TaxPraBen, a comprehensive benchmark designed to evaluate LLMs across three cognitive levels based on Bloom's Taxonomy: Knowledge Memorization (KM), Knowledge Understanding (KU), and Knowledge Application (KA).

A. Data Construction

TaxPraBen comprises 14 datasets with 7,300 instances, constructed through a rigorous multi-source fusion strategy:

1. Sources: Data is aggregated from tax exam guidebooks, official documents (State Administration of Taxation), online tax news/risk reports, and public tax case judgments.
2. Processing Pipeline:
   • Collection: OCR and text reproduction from books; API downloads from official sites; web scraping for news and cases.
   • Augmentation: ChatGPT is used for text compression, Q&A pair generation, and structured element extraction.
   • Human-in-the-Loop: A dedicated team of 50 annotators (experts in finance, law, and CS) performs manual verification, correction, and "Availability Checks" to eliminate hallucinations and ensure factual accuracy.
3. Task Categories:
   • KM (200-1000 samples): Tax law recitation (TaxRecite).
   • KU (1000 samples): Summarization, topic classification, and reading comprehension.
   • KA (1000+ samples): The core innovation, featuring 10 real-world scenarios including tax calculation, single/multiple-choice exams, Q&A, legal identification, risk prevention, inspection analysis, and tax strategy planning.

B. Structured Evaluation Paradigm

To address the challenge of evaluating hybrid text-numeric outputs, TaxPraBen proposes a Structured Evaluation Pipeline:

1. Unified Output Protocol: Models are prompted to generate responses in a strict JSON format.
2. Parsing Layer: Since open-source models often fail to adhere to strict formatting, ChatGPT-3.5 is employed as a "structure-aware parser" to normalize raw outputs, extract key fields, and convert them into valid JSON.
3. Dual-Dimension Metrics:
   • Semantic Evaluation: Uses BERTScore and BARTScore to assess the quality of textual reasoning and explanations.
   • Numerical Evaluation: Uses Exact Match (EM) Accuracy for critical numerical fields (e.g., tax amounts, penalties).
   • Aggregation: Final scores (e.g., PlanScore, InspectScore) are weighted averages of semantic and numerical metrics, ensuring both reasoning logic and calculation accuracy are penalized if incorrect.

3. Key Contributions

1. First Chinese Tax Practice Benchmark: TaxPraBen is the first benchmark specifically designed for real-world Chinese tax scenarios, moving beyond isolated NLP tasks to include complex applications like risk prevention, inspection analysis, and tax planning.
2. High-Quality, Expert-Annotated Dataset: It fills the data gap with 7.3K manually verified instances, curated by domain experts and enhanced with LLM-assisted annotation, ensuring high fidelity to actual tax workflows.
3. Bloom's Taxonomy Framework: The benchmark organizes tasks into a hierarchical cognitive structure (Memorization $\to$ Understanding $\to$ Application), providing a nuanced view of model capabilities.
4. Scalable Structured Evaluation Method: The proposed "structured parsing—field alignment—matching" paradigm is domain-agnostic and can be extended to other fields requiring hybrid outputs (e.g., law, finance, healthcare).
5. Comprehensive Evaluation: It evaluates 19 representative LLMs (including closed-source giants like GPT-4o and ERNIE-3.5, and various Chinese open-source models) under both zero-shot and one-shot settings.

4. Key Results

The evaluation of 19 LLMs reveals significant performance disparities and specific weaknesses:

• Closed-Source Dominance: Large-parameter, closed-source models (ERNIE-3.5, GPT-4o, Grok3) consistently outperform others, highlighting the value of massive parameters and domain-specific knowledge integration.
• Chinese Model Advantage: Chinese-oriented LLMs (e.g., Qwen2.5, DeepSeek) generally outperform multilingual models on tasks involving Chinese tax terminology and policies.
• Fine-Tuning Limitations: YaYi2, a model fine-tuned on tax data, showed limited improvement and often underperformed compared to general-purpose models. This suggests that without sufficient data diversity and task alignment, domain fine-tuning alone is insufficient.
• Application vs. Memorization: Performance drops significantly from Knowledge Memorization to Knowledge Application. Models struggle most with TaxPlan (strategy planning) and TaxCalc (complex calculations), indicating a weakness in multi-step numerical reasoning and logical integration.
• One-Shot Effects: While some models improved with one-shot examples, others (like GLM4) declined, suggesting that prompt design and model architecture heavily influence few-shot learning stability.
• Reasoning Gap: All models, even top-tier ones, struggle with complex reasoning tasks requiring both semantic understanding and precise calculation, revealing a fundamental gap in current LLMs for professional tax practice.

5. Significance

• Practical Relevance: TaxPraBen shifts the evaluation paradigm from "can the model answer a question?" to "can the model solve a real-world tax problem?" This is crucial for deploying AI in government and enterprise tax systems.
• Reliability Assessment: By exposing the inability of current models to handle complex numerical reasoning and structured outputs, the benchmark prevents the overestimation of LLM capabilities in high-stakes domains.
• Future Research Direction: The benchmark provides a vital resource for developing specialized tax LLMs and offers a scalable evaluation framework that can be adapted for other regulated industries (Law, Medicine, Finance) where accuracy and structure are paramount.
• Open Resource: The authors provide an open leaderboard and dataset, fostering transparency and community-driven improvement in Chinese AI for tax practice.