Technical Summary: Adversarial Alignment for Value Consistency in LLMs

Problem Statement

The rapid proliferation of Large Language Models (LLMs) has introduced significant societal risks, particularly regarding value biases and inconsistencies in sensitive domains such as race, religion, politics, and sovereignty. While existing alignment methods focus on general safety and fairness, they often neglect the specific alignment of values within these sensitive contexts. Current models frequently exhibit biases stemming from training data, display reluctance to answer political questions, or provide value-inconsistent responses. There is a notable lack of dedicated alignment methods, training data, and evaluation benchmarks specifically designed for sensitive domains.

Methodology

The authors propose an Adversarial Alignment Framework designed to enhance value consistency through a three-stage training pipeline:

1. Continued Pre-training: The model undergoes continued pre-training on value-aligned data sourced from sensitive domains (Sovereignty, Human Rights, Religion, Politics, and Ethnicity) to mitigate biases inherent in original pre-training corpora.
2. Instruction Fine-tuning: Two specific instruction formats are employed:
   • Question-Answering: Generating value-aligned answers to direct queries.
   • Statement-Rebuttal: Identifying value-inconsistent statements and producing rebuttals that reflect aligned values.
3. Adversarial Training: This core component utilizes a collaborative triad of models to generate high-quality, challenging training data:
   • Attacker: A fine-tuned model (Qwen2.5-7B) that generates controversial or biased queries and statements in sensitive domains.
   • Actor: A more capable model (Qwen2.5-72B) that generates responses adhering to value alignment principles, either answering queries truthfully or rebutting inconsistent statements.
   • Critic: An independent evaluator (GLM-4-9B) that filters the generated responses, ensuring they are not evasive, incomplete, or value-misaligned.

Based on this framework, the authors trained VC-LLM, a value-consistent large language model specifically for Chinese sensitive domains. They also constructed a bilingual evaluation dataset (Chinese and English) covering 31 subcategories across the five sensitive domains.

Key Contributions

• Adversarial Alignment Framework: A novel method to automatically generate high-quality, challenging alignment datasets for sensitive domains using an Attacker-Actor-Critic architecture.
• VC-LLM: The training and public release of a large language model specifically optimized for value consistency in Chinese sensitive domains, alongside the associated training dataset.
• Bilingual Evaluation Benchmark: The construction of a comprehensive Chinese-English evaluation benchmark to assess value alignment in sensitive domains, facilitating comparative analysis of mainstream LLMs.

Experimental Results

The authors evaluated VC-LLM against various mainstream baselines (including Qwen2.5, Baichuan2, GLM-4, Llama-3, and GPT-4o) on their bilingual benchmark.

• Performance: VC-LLM outperformed all baseline models in both Chinese and English evaluations. In Chinese, it achieved a total score of 836 (average 4.80/5.00), surpassing the second-best model (Qwen2.5-72B-Instruct) by 0.40 points. In English, it scored 799 (average 4.59), nearly 1.0 point higher than the next-best model.
• Domain Specifics: VC-LLM showed particularly strong performance in Religion and Politics. While it performed well in Sovereignty, this domain presented more challenges, likely due to the complexity and cross-cultural sensitivity of the content.
• Impact of Adversarial Training: Ablation studies confirmed that adversarial training significantly improved alignment. Removing this stage resulted in a substantial drop in scores (e.g., a drop of 1.08 in Chinese average and 2.53 in English average for VC-LLM). Applying the same adversarial training to a general model (Qwen2.5-7B) also yielded significant improvements, demonstrating the method's generalizability.
• Bilingual Consistency: VC-LLM exhibited a smaller performance gap between Chinese and English compared to other models, indicating more balanced bilingual value alignment.

Significance and Claims

The paper claims that the proposed adversarial alignment framework effectively addresses the gap in value consistency for LLMs in sensitive domains. By integrating continued pre-training, instruction fine-tuning, and a multi-agent adversarial training process, the authors demonstrate that it is possible to train models that maintain strong value alignment without sacrificing performance. The results suggest that mainstream LLMs currently suffer from value inconsistencies in these domains due to conflicting signals in their training data, and that targeted adversarial alignment can significantly mitigate these issues. The authors conclude that their approach offers a viable path toward more reliable and value-consistent AI systems for sensitive applications.