SteuerLLM: Local specialized large language model for German tax law analysis

Technical Summary: SteuerLLM and SteuerEx

Problem Statement
Large language models (LLMs) have demonstrated robust general reasoning capabilities but remain fragile in domains governed by strict formal rules, precise terminology, and legally binding structures, such as tax law. In these fields, correct reasoning requires exact statutory citation, hierarchical interpretation of interdependent norms, structured argumentation, and numerical accuracy. Minor errors can invalidate otherwise plausible answers. Existing legal AI benchmarks often rely on synthetic datasets, short-answer formats, or binary correctness metrics that fail to capture the graded, partial-credit nature of real-world legal examinations. Furthermore, no open benchmark derived from authentic German university tax law examinations has been available to rigorously evaluate domain-specific reasoning across multiple tax domains and academic levels.

Methodology
The study introduces two primary components: the SteuerEx benchmark and the SteuerLLM model.

• SteuerEx Benchmark: This is the first open benchmark constructed from 115 authentic German university tax law examinations administered between 2016 and 2024 at Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU). The benchmark covers six core domains: corporate tax, fiscal code, fundamentals of tax law, income tax, partnership taxation, and value-added tax (VAT). Unlike standard benchmarks, SteuerEx employs a statement-level, partial-credit evaluation framework. Expert-validated reference solutions are decomposed into discrete, independently scorable legal statements with assigned point values. This design mirrors real academic grading, allowing for the assessment of incremental legal reasoning quality rather than just final correctness.
• SteuerLLM Architecture and Training: SteuerLLM is a domain-adapted LLM (28B parameters) built upon the Mistral Small 2409 (24B) and Ministral (8B) base architectures. To incorporate domain-specific capacity without degrading general reasoning abilities, the authors employ a block expansion strategy. This involves inserting additional trainable Transformer layers into the frozen pretrained backbone, allowing the model to specialize through increased depth rather than full fine-tuning.
• Synthetic Data Generation: Due to the scarcity of large-scale, structured German tax law datasets, the authors developed a controlled retrieval-augmented pipeline called the Water Fountain Algorithm. Starting from a seed set of ~1,800 authentic exam questions, the algorithm iteratively retrieves relevant statutory texts and commentaries using a local SearXNG instance. It then generates new, thematically diversified question-answer pairs using a large language model, ensuring legal precision and context sufficiency. The final training dataset comprises 485,092 validated question-answer pairs.
• Evaluation Protocol: Models were evaluated on SteuerEx using a statement-level grading pipeline where an external LLM (GPT-4o) acts as an evaluator, assessing conceptual correctness, statutory accuracy, and completeness against the reference statements. The evaluation excludes retrieval augmentation at inference time to isolate internalized knowledge. Statistical significance was assessed using paired permutation tests and non-parametric bootstrapping.

Key Contributions

1. SteuerEx Benchmark: An open, expert-validated benchmark of 115 authentic exam questions with a granular, partial-credit scoring system that reflects real academic assessment practices in German tax law.
2. SteuerLLM: A domain-specialized 28B parameter model trained on a large-scale synthetic dataset derived from authentic examinations, utilizing a block expansion architecture to balance specialization with general capabilities.
3. Open-SteuerLLM: A variant of the model trained without a private subset of data, released to demonstrate the trade-offs between data openness and performance in domain-specialized models.
4. Comprehensive Evaluation: A systematic comparison of SteuerLLM against a diverse spectrum of instruction-tuned and reasoning-oriented LLMs (ranging from 3B to 671B parameters), including proprietary systems like GPT-4o-mini.

Results

• Performance vs. Scale: On the SteuerEx benchmark, general-purpose instruction-tuned models (even those with up to 72B parameters) scored significantly lower than SteuerLLM. For instance, SteuerLLM (28B) achieved 28% ± 2, outperforming Qwen2.5-72B-it (19% ± 3) and GPT-4o-mini (22% ± 2). The only model to outperform SteuerLLM was DeepSeek-R1-671B (39% ± 3), a reasoning-specialized model with 24× the parameter count.
• Domain Specificity: The results indicate that parameter scale alone does not reliably predict performance in tax law. Domain-specific training and architectural adaptation were more decisive than raw parameter count. A smaller domain-adapted variant, Small-SteuerLLM (10B), achieved 16% ± 2, performing competitively with general-purpose models in the 14B–32B range.
• Human Comparison: While SteuerLLM did not reach the average performance of human students (who averaged 54–63% across categories), it exceeded the lowest observed student grades in several domains (e.g., corporate tax, VAT, and fundamentals of tax law). This suggests the model can produce partially correct, exam-relevant reasoning rather than merely generating superficially plausible answers.
• Open vs. Closed Data: Open-SteuerLLM (trained without private data) achieved 23% ± 3, a reduction of approximately 5 percentage points compared to the full SteuerLLM. This difference was not statistically significant at the aggregate benchmark level, though performance gaps were more pronounced in smaller, specialized categories.

Significance and Claims
The paper claims that realistic academic assessments, such as university tax law examinations, serve as high-fidelity benchmarks for evaluating the generalization and specialization of deep learning systems. The study demonstrates that for structured legal reasoning tasks, domain-specific data and architectural adaptation are more decisive than parameter scale alone.

The authors emphasize that SteuerEx provides a rigorous framework for studying legal AI by operationalizing evaluation through structured reasoning, statutory precision, and incremental partial credit. This approach reveals capability differences that are often obscured by conventional legal NLP benchmarks. Furthermore, the work highlights that while domain adaptation yields substantial gains, significant gaps remain between current LLMs and human experts, particularly in highly specialized or procedurally complex domains. The release of the benchmark, training data, and model weights aims to support reproducible research in domain-specific legal artificial intelligence.