Technical Summary: YouZhi-LLM

Problem Statement

Large Language Models (LLMs) have driven significant innovation in digital finance, yet their deployment in real-world, high-concurrency scenarios (e.g., mobile banking) faces a critical bottleneck: KV cache memory overhead. While existing financial LLMs (e.g., BloombergGPT, FinGPT) improve domain knowledge through pre-training or supervised fine-tuning (SFT), they typically retain standard Grouped-Query Attention (GQA) or Multi-Head Attention (MHA) architectures. These structures suffer from high memory consumption during inference, limiting scalability and increasing infrastructure costs. Furthermore, existing approaches often treat model capability and serving efficiency as isolated concerns, failing to address the structural disconnect between theoretical performance and production deployability.

Methodology

The authors propose YouZhi-LLM, a high-efficiency financial LLM built natively on the Huawei Ascend ecosystem. The solution comprises three core components:

1. Layer-Adaptive GQA-to-MLA Transition

To reduce KV cache overhead, the paper transitions the model architecture from GQA to Multi-Head Latent Attention (MLA). However, naive conversion (e.g., using the TransMLA framework with a uniform configuration) causes significant perplexity degradation.

• Observation: The authors discovered that different transformer layers exhibit diametrically opposed degradation characteristics during the transition. Shallow layers benefit from larger compression (larger FreqFold sizes), while middle layers require minimal compression (smaller FreqFold sizes) to preserve Rotary Position Embedding (RoPE) frequency information.
• Algorithm: Instead of applying a uniform FreqFold size across all layers, the authors propose a layer-adaptive transition framework. They formulate the selection of FreqFold sizes as a combinatorial optimization problem, which is solved via a multi-round sequential decision process. This dynamically assigns optimal FreqFold sizes per layer, maximizing KV-cache compression while minimizing perplexity degradation.

2. Comprehensive Post-Training Pipeline

To recover representation capacity lost during the structural transition and inject domain expertise, a two-stage post-training pipeline is implemented on Ascend clusters:

• Stage 1: Generalized Knowledge Distillation (GKD): The original GQA model serves as a teacher to the converted MLA student model. Using forward KL divergence and an off-policy strategy on mixed offline datasets, this stage restores general language modeling capabilities degraded by the architecture change.
• Stage 2: Financial Domain-Specific SFT: The model is fine-tuned on a curated dataset of 970,000 high-quality instruction-response pairs covering major financial sub-domains.

3. Advanced Data Construction

The training data pipeline is designed to address specific challenges introduced by the structural transition and domain requirements:

• Stratified Data Compression: A three-stage method (tagging, stratified filtering, intra-cluster entropy compression) balances general and financial data, ensuring category completeness and difficulty balance while reducing redundancy.
• Domain Augmentation: Missing capabilities (e.g., specific financial reasoning, refusal handling for incomplete data) are addressed via progressive diversity data generation, logic tree modeling for unanswerable questions, and Self-Instruct/Evol-Instruct for capability-specific data amplification.
• Instruction-Following: Specific templates are used to reinforce the model's ability to output structured formats (JSON, Markdown) required in financial applications.

Key Contributions

1. Layer-Wise Analysis: The paper reveals that shallow and deep layers exhibit distinct degradation patterns during GQA-to-MLA transitions, challenging the assumption that uniform conversion strategies are optimal.
2. Layer-Adaptive Algorithm: A novel algorithm that dynamically assigns per-layer FreqFold sizes, reducing perplexity degradation by up to 35% compared to uniform baselines (TransMLA) across multiple LLM backbones.
3. Integrated Training Pipeline: A comprehensive post-training strategy combining GKD and financial SFT, specifically tailored for converted MLA models, which recovers lost capabilities and enhances financial performance.
4. Ascend-Native Deployment: Implementation of YouZhi-LLM on Huawei Ascend NPUs using the vLLM-Ascend framework, demonstrating the practical viability of the approach in production environments.

Experimental Results

Evaluations were conducted on Huawei Ascend A3 clusters using the vLLM-Ascend inference framework.

• Language Modeling: The layer-adaptive approach significantly reduced perplexity degradation. For example, on Llama3-8B, the degradation was reduced by 65% compared to the TransMLA baseline.
• Financial Benchmarks:
  • YouZhi-7B (based on OpenPangu-7B) achieved a 12.3% improvement in average financial benchmark scores compared to its base model.
  • YouZhi-14B (based on Qwen2.5-14B) achieved a 7.0% accuracy gain.
  • Both models outperformed strong financial baselines (e.g., YiZhao-12B-Chat, DianJin-R1-7B) and recent general models (Qwen3/3.5).
• High-Concurrency Efficiency:
  • KV Cache Reduction: YouZhi-7B reduced KV cache size by 72% (from 2,048 to 576 elements).
  • Concurrency: This reduction translated to a 2.69× increase in maximum concurrent requests for YouZhi-7B (from 95 to 256) and a 2.43× increase for YouZhi-14B.
  • Throughput: Peak throughput increased by 1.76× for YouZhi-7B.
• Real-World Application: In mobile banking scenarios (intent recognition and slot filling), YouZhi-7B maintained performance comparable to its base model (average accuracy ~97.7% for intent, ~99.0% for slot filling) while offering superior concurrency.

Significance

The paper claims to establish a new paradigm for cost-effective, high-throughput financial inference. By bridging the gap between architectural efficiency (via adaptive MLA transition) and domain capability (via targeted post-training), YouZhi-LLM demonstrates that it is possible to achieve the "Pareto frontier" of both high accuracy and high concurrency. The work validates a pathway for deploying industry-specific LLMs natively on the Huawei Ascend ecosystem, addressing the critical operational constraints of low latency, high concurrency, and high task-completion rates required by real-world financial services.