KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge

Technical Summary: KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge

Problem Statement
While Large Language Models (LLMs) have demonstrated impressive global capabilities, their deployment in specific nations is hindered by a lack of alignment with local cultural contexts. Existing alignment research predominantly focuses on universal moral principles or Western-centric perspectives, failing to account for the fact that human values and the importance of specific knowledge vary significantly across cultures, countries, and time periods. Current datasets often exhibit a bias toward English-speaking cultures and lack the granularity to measure how well a model reflects the general opinions and basic knowledge of a specific nation, such as South Korea. Consequently, there is a need for a benchmark that evaluates "National Alignment," defined as the degree to which an LLM aligns with a targeted country's social values and common knowledge.

Methodology
The authors introduce KorNAT (Korean National Alignment Test), the first benchmark designed to measure national alignment with South Korea. The dataset comprises 10,000 multiple-choice questions divided into two distinct categories:

1. Social Value Dataset (4,000 samples):

   • Topic Selection: Keywords were extracted from two sources: monthly social conflict reports (covering gender, age, and wealth gaps) and timely news articles from the 12 months prior to construction.
   • Question Generation: Initial questions were generated using GPT-3.5-Turbo based on the keywords and associated news articles, then refined through two rounds of human review by college-educated workers to ensure relevance to current Korean social values.
   • Ground Truth Construction: Instead of assigning arbitrary labels, the ground truth was derived from a large-scale survey of 6,174 unique Korean participants (aged 19+). Participants selected from five options (Strongly Disagree to Strongly Agree).
   • Statistical Rigor: The survey employed stratification and sampling theory to ensure representativeness. To address demographic imbalances in the online survey pool (e.g., under-representation of those over 60), the authors applied Stratification Adjustment and Sampling Adjustment weights based on age, gender, education, residence, and income. This process yielded an average of 219 responses per question, achieving a 5.5% error bound.
   • Quality Control: Distractor questions and consistency checks were embedded in the survey to filter out low-quality responses.

2. Common Knowledge Dataset (6,000 samples):

   • Source Material: Questions were constructed based on Korean compulsory education textbooks and GED (General Educational Development) reference materials covering seven subjects: Korean, Social Studies, Korean History, Common Sense, Mathematics, Science, and English.
   • Construction: Human workers (college graduates or higher) paraphrased content from reference materials into four-choice questions. Complex samples required multi-hop reasoning (combining two facts), while simple samples required single facts.
   • Quality Control: Two rounds of human revision ensured grammatical correctness, standardized answer lengths to prevent bias, and verified factual accuracy against source materials. No language models were used in the generation of these specific questions to avoid AI-induced errors.

Evaluation Metrics

• Social Value Alignment: Since social values are subjective, the authors propose three metrics based on the distribution of survey responses rather than a single "correct" answer:
  • SVA (Social Value Alignment): Measures the probability of the model's predicted option matching the survey distribution.
  • A-SVA (Aggregated SVA): Aggregates "Strongly Disagree/Disagree" and "Strongly Agree/Agree" into three categories to increase the maximum achievable score.
  • N-SVA (Neutral-processed SVA): Treats "Neutral" as a one-hot distribution if no other option surpasses a 0.5 threshold, penalizing models that fail to take a stance on polarized issues.
• Common Knowledge Alignment: Measured using standard accuracy, with a reference score of 0.6 (60%) based on the Korean GED passing threshold.

Experimental Results
The authors evaluated seven major LLMs (Llama-2, GPT-3.5-Turbo, GPT-4, Claude-1, HyperCLOVA X, PaLM-2, and Gemini Pro) on KorNAT.

• Social Values: Only two models exceeded the reference score in SVA and A-SVA. PaLM-2 generally achieved the highest alignment scores, while HyperCLOVA X (a Korean-specific model) performed best in N-SVA, suggesting a tendency to avoid polarized topics. Cross-national prompting (asking how a Korean would respond) generally improved alignment scores.
• Common Knowledge: Only three models surpassed the 0.6 reference score. HyperCLOVA X demonstrated outstanding performance, particularly in Korean language, history, and social studies, significantly outperforming other models in subjects requiring deep cultural context. However, even the best-performing models only marginally exceeded the reference score, indicating a general gap in Korean common knowledge alignment.
• Small Models: Experiments with smaller, fine-tuned Korean models (e.g., KoLlama-2) showed that additional fine-tuning on Korean corpora improves social value alignment compared to multilingual base models, though none met the 0.6 baseline for common knowledge.

Key Contributions

1. Introduction of National Alignment: The paper defines and operationalizes "National Alignment" as a distinct concept from global value alignment, focusing on the intersection of social values and common knowledge specific to a nation.
2. KorNAT Benchmark: The creation of a high-quality, 10K-sample dataset for South Korea, featuring a meticulously curated social value dataset grounded in statistical sampling theory and a common knowledge dataset based on national education standards.
3. Government Approval: The dataset underwent rigorous qualitative and quantitative assessment by the Telecommunications Technology Association (TTA), a government-affiliated organization, earning official approval for dataset quality.
4. Evaluation Framework: The proposal of three distinct metrics (SVA, A-SVA, N-SVA) to capture the nuance of social value alignment, acknowledging that a single majority vote may not fully represent the distribution of public opinion.

Significance
The authors assert that for LLMs to be effectively deployed in a specific country, they must possess an understanding of that nation's culture and basic knowledge. KorNAT provides the first standardized framework to measure this alignment for South Korea. The results highlight that current state-of-the-art models, even those trained on massive corpora, are not sufficiently aligned with South Korean social values and common knowledge, underscoring a significant area for improvement. The paper emphasizes that while the dataset is currently specific to Korea, the creation framework is generalizable and intended to inspire researchers to develop similar benchmarks for other nations to foster a more inclusive understanding of diverse national characteristics in AI.