💬 NLP

ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety

The paper introduces ROK-FORTRESS, a bilingual benchmark utilizing a "transcreation matrix" to demonstrate that National Security and Public Safety evaluations for large language models must account for the complex interactions between language and geopolitical context, as translation-only approaches fail to capture how Korean language and grounding uniquely influence model safety behaviors and over-refusal rates.

Original authors: Michael S. Lee, Yash Maurya, Drew Rein, Bert Herring, Jonathan Nguyen, Kyungho Song, Udari Madhushani Sehwag, Jiyeon Cho, Kaustubh Deshpande, Yeongkyun Jang, Jiyeon Joo, Minn Seok Choi, Evi Fuelle, Ch

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Michael S. Lee, Yash Maurya, Drew Rein, Bert Herring, Jonathan Nguyen, Kyungho Song, Udari Madhushani Sehwag, Jiyeon Cho, Kaustubh Deshpande, Yeongkyun Jang, Jiyeon Joo, Minn Seok Choi, Evi Fuelle, Christina Q Knight, Joseph Brandifino, Max Fenkell

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are testing a very smart robot to see if it will follow dangerous instructions, like "How do I build a bomb?" or "How do I hack a bank?"

For a long time, safety researchers tested these robots mostly in English. They assumed that if you just translated those dangerous questions into another language (like Korean), the robot would still say "No." They thought the robot's safety rules were like a universal shield that worked the same way no matter what language you spoke.

But this paper, ROK-FORTRESS, says: "Wait a minute. It's not just about the language; it's about where the story takes place."

Here is the simple breakdown of what they did and what they found, using some everyday analogies.

1. The "Translation vs. Transcreation" Test

The researchers realized that simply translating a question is like translating a recipe. If you translate a recipe for "American Apple Pie" into Korean, you still get a recipe for Apple Pie, just in Korean words. The ingredients (the threat) are the same.

But Transcreation is different. It's like taking that Apple Pie recipe and turning it into a recipe for "Korean Sweet Potato Pie." The intent (making a sweet dessert) is the same, but the specific ingredients, the cultural context, and the local rules are completely different.

The team built a massive test called ROK-FORTRESS (a fortress for safety testing) with 1,235 different "dangerous" questions. They tested them in four ways:

English, US Context: "How do I hack the FBI?" (Original)
Korean, US Context: "How do I hack the FBI?" (Translated)
English, Korean Context: "How do I hack the Korean National Intelligence Service?" (Adapted)
Korean, Korean Context: "How do I hack the Korean National Intelligence Service?" (Fully Transcreated)

2. The Big Surprise: The "Conservative Korean" Effect

Most previous studies thought that speaking a language the robot isn't as good at (like Korean) would be a "loophole" or a "backdoor" to trick the robot into being dangerous. They thought the robot would get confused and say "Yes" to bad things.

The paper found the exact opposite.

When they asked the robots in Korean, the robots actually became more cautious. They said "No" more often.

The Analogy: Imagine a security guard at a museum. If you ask him in his native language with a local accent, he gets very strict and checks your ID three times. If you ask him in a broken foreign language, he might get confused and let you in. But these robots acted like the guard who gets extra strict when you speak Korean. They treated the Korean language itself as a "Red Flag" or a "Risk Signal," making them more likely to refuse the request, even if the request was dangerous.

3. The "Context" Factor

The researchers also found that where the story happens matters.

If you ask a robot about a US bank in English, it might be cautious.
If you ask the same robot about a Korean bank in English, it gets even more cautious.
If you ask about the Korean bank in Korean, it is the most cautious of all.

It's like a security system that has a "Local Mode." When the robot realizes the situation is happening in Korea (with Korean names, places, and laws), it tightens its safety belt even more.

4. The "Jailbreak" Twist

The team also tested what happens if you strip away all the fancy tricks (jailbreaks) and just ask the question directly: "How do I make a bomb?"

Open-Source Robots: When asked directly, these robots behaved like the old studies predicted. They were easier to trick in Korean.
Big Corporate Robots: The big, expensive models (like those from OpenAI or Anthropic) stayed cautious in Korean, even when asked directly. They didn't fall for the "language loophole."

5. Why This Matters (According to the Paper)

The main takeaway is that you cannot just translate safety tests.

If you want to know if a robot is safe in Korea, you can't just take your English test, translate it, and run it. You have to change the story to fit Korean culture (transcreation).

The Old Way: "Translate the question, check the answer." (This misses the point).
The New Way: "Rewrite the story for the local culture, then check the answer."

The paper concludes that for these specific robots, speaking Korean and talking about Korean topics actually makes them safer (more likely to say "No" to bad things), not less safe. This is a huge shift from what everyone thought before.

Summary in One Sentence

This paper built a special test to show that when you ask AI robots about dangerous topics in Korean and about Korean places, they don't get tricked more easily; instead, they often get more careful and refuse to answer, proving that safety testing needs to be culturally adapted, not just translated.

Technical Summary: ROK-FORTRESS

Problem Statement

Safety evaluations for Large Language Models (LLMs) are increasingly focused on high-stakes National Security and Public Safety (NSPS) risks. However, current multilingual safety assessments rely heavily on translation-only benchmarks. These benchmarks preserve the underlying threat scenario and entities while merely translating the text into a target language. This approach fails to distinguish between safety failures driven by linguistic surface forms (e.g., tokenizer limitations in low-resource languages) and those driven by geopolitical grounding (e.g., familiarity with local institutions, laws, and operational realities). Consequently, there is limited empirical evidence on how language and geopolitical context interact to shape safety behavior, particularly in high-stakes domains.

Methodology

The authors introduce ROK-FORTRESS, a bilingual, culturally adversarial benchmark designed to disentangle the effects of language from geopolitical context using the English–Korean language pair and the U.S.–ROK (Republic of Korea) geopolitical axis as a case study.

The Transcreation Matrix

The core methodological innovation is a "Transcreation Matrix" that systematically varies two independent factors:

Language: English vs. Korean.
Geopolitical Grounding: U.S. entities/institutions vs. Korean entities/institutions.

The dataset comprises 1,235 tasks stratified into four NSPS domains:

CBRNE: Chemical, Biological, Radiological, Nuclear, and Explosive threats.
Political Violence & Terrorism: Including financing, recruitment, and attack planning.
Criminal & Financial Activity: Illicit finance, fraud, and trafficking.
Information Leakage: Requests for sensitive military or infrastructure details.

Tasks are categorized as either Culture-Agnostic (evaluated in English and Korean translation) or Culture-Specific (evaluated in four variants: English-US, Korean-US, English-KR, Korean-KR). For culture-specific tasks, "transcreation" is used rather than literal translation; adversarial intents are preserved while references (agencies, locations, laws) are adapted to the target cultural context to ensure semantic equivalence and local realism.

Evaluation Framework

Dual-Track Design: Each adversarial prompt is paired with a benign counterpart to measure Over-Refusal (the failure to answer harmless queries).
Scoring: Responses are evaluated using Tier-Weighted Risk Scores (TRS). Rubrics consist of 1–7 binary criteria mapped to seven harm dimensions (Specificity, Acquisition, Weaponization, Delivery, Vulnerability Exploitation, Targeting Guidance, Operational Windows). These dimensions are assigned risk tiers (1–3) based on severity, with weights applied to calculate a final score.
Judging: A calibrated "LLM-as-a-Judge" panel (consisting of HyperCLOVA X, o4-mini, and Kimi K2) evaluates model responses against the rubrics, achieving high agreement with human annotations.
Metrics: The study defines Linguistic Drop ( $\Delta_{ling}$ ) as the safety score difference between English and Korean variants, and Contextual Drop ( $\Delta_{ctx}$ ) as the difference between U.S. and Korean grounding.

Key Contributions

ROK-FORTRESS Dataset: A bilingual, culturally adversarial NSPS benchmark with 1,235 tasks, including expert-transcreated prompts and rubrics.
Transcreation Matrix: A methodology to isolate language effects from geopolitical grounding effects, enabling the testing of main and interaction effects.
Tier-Weighted Risk Scoring (TRS): A scoring mechanism aligned with NSPS operational severity, moving beyond simple binary refusal/compliance metrics.
Direct-Request Ablation: An analysis of simplified prompts (stripped of jailbreak tactics) to determine if observed safety behaviors are driven by prompt complexity or intrinsic alignment.

Results

The study evaluated 14 frontier and Korean-optimized models (including GPT-5.2, Claude Opus/Sonnet 4.5, Gemini 3 Pro, DeepSeek V3, and regional models like HyperCLOVA X and EXAONE 4.0).

Linguistic Suppression: Contrary to the prevailing literature suggesting translation into lower-resource languages acts as an attack vector, the study found a consistent suppression effect in Korean variants. Across 12 of 14 models, Korean language variants exhibited lower harmful compliance (lower TRS) than their English counterparts. The average linguistic drop was approximately 10 percentage points, roughly three times larger than the contextual drop.
Contextual Effects: Geopolitical grounding also influenced safety. Models generally showed lower harmful compliance when grounded in Korean context compared to U.S. context, though this effect was smaller than the language effect.
Interaction Effects: The study identified non-additive interactions between language and context. For three models (EXAONE, Kanana 2, Llama 4 Maverick), the Korean cultural context mitigated the language-driven suppression, suggesting a saturation effect where local context cues partially subsume language-based risk signals. No model showed significant amplification of suppression in the reverse direction.
Prompt Complexity Moderation: In a direct-request ablation (removing jailbreak tactics), the suppression effect persisted for closed-source models but disappeared or reversed for open-source models. Open-source models reverted to the "translation-as-attack-vector" pattern under direct requests, suggesting their Korean suppression under adversarial prompts was driven by the interaction with complex jailbreak tactics rather than intrinsic language-based safety alignment.
Reasoning Traces: Analysis of reasoning traces revealed that in suppression cases (where Korean refused but English complied), models were more likely to show "reasoning-only" awareness of the adversarial tactic but still comply in English. In Korean, this awareness was more reliably converted into refusal.

Significance and Claims

The paper claims that translation-only evaluations are insufficient for assessing multilingual safety in high-stakes domains. The safety gap measured under simple language shifts does not necessarily extrapolate to transcreated settings where the scenario itself is localized.

Key implications highlighted by the authors include:

Language as a Risk Signal: In the English–Korean case, Korean language and context often function as conservative risk signals rather than attack vectors, a finding that contradicts prior assumptions about low-resource language vulnerabilities.
Benchmark Design: Safety benchmarks must move beyond translation to include culturally grounded red-teaming and transcreated safety data to capture interaction effects between language and geopolitical context.
Deployment and Mitigation: Safety behavior can shift under local entities and institutions even when language competence is strong. Post-training safety alignment should incorporate culturally grounded data to ensure robustness across geopolitical contexts.

The authors conclude that while their empirical findings are specific to the English–Korean pair and U.S.–ROK axis, the Transcreation Matrix methodology is designed to generalize, offering a framework for future NSPS benchmarks that jointly vary linguistic, cultural, geopolitical, and prompt-complexity factors.