High-Risk Memories? Comparative audit of the representation of Second World War atrocities in Ukraine by generative AI applications

1. Problem Statement

The paper addresses the emerging risk that generative AI (genAI) applications pose to the accuracy and ethical representation of high-risk memories—specifically, emotionally charged and politically contested historical events like mass atrocities.

• The Core Issue: GenAI models (e.g., ChatGPT, Gemini, Copilot) operate on probabilistic token prediction rather than factual understanding. This makes them susceptible to hallucinations (fabricating facts) and inconsistent moralization (applying selective normative judgments).
• The Context: These risks are amplified for "high-risk memories" (e.g., WWII atrocities in Ukraine) which are often instrumentalized for political propaganda (e.g., by the Kremlin) and lack a unified historical consensus across different linguistic and national communities.
• The Gap: While scholarship exists on genAI and history, there is a lack of empirical, comparative data on how different genAI models handle specific, contested atrocities across multiple languages (English, Ukrainian, Russian).

2. Methodology

The authors conducted a comparative audit using a structured prompt engineering approach.

• Subjects: Three major genAI applications were tested:
  1. Bing Chat (now Copilot, Microsoft)
  2. Google Bard (precursor to Gemini, Google)
  3. ChatGPT (OpenAI)
• Dataset:
  • Prompts: 74 distinct prompts focusing on WWII atrocities in Ukraine.
  • Topics: The Holocaust, atrocities against Poles, and atrocities against Ukrainians (including sensitive topics regarding the Organization of Ukrainian Nationalists [OUN], Stepan Bandera, and collaborationist units).
  • Languages: Prompts were translated and tested in English, Ukrainian, and Russian.
  • Total Output: 666 responses (74 prompts × 3 models × 3 languages).
• Baseline for Evaluation: A "human baseline" was established using fact-based interpretations from Western and Ukrainian Holocaust experts, excluding ideology-driven claims.
• Analysis Framework:
  • Accuracy: Responses were coded as fully accurate, partially accurate, or inaccurate based on alignment with the human baseline.
  • Hallucinations: Identification of fabricated details, dates, numbers, or non-existent testimonies.
  • Moralization: Detection of normative statements (e.g., "dark chapter," "horrific," "important to remember") and their consistency across languages and models.
• Reliability: Two coders analyzed the data, achieving a high intercoder agreement (Krippendorf's alpha of 0.93).

3. Key Contributions

• Empirical Audit of High-Risk Memory: This is one of the first studies to systematically audit genAI performance specifically on contested WWII atrocities in Ukraine, a region where memory politics are highly volatile.
• Multilingual Disparity Analysis: The study highlights significant performance gaps between languages, demonstrating that "low-resource" languages (Ukrainian, Russian) suffer from higher rates of inaccuracy and hallucination compared to English.
• Taxonomy of Misrepresentation: The paper distinguishes between two distinct failure modes:
  1. Factual Distortion: Hallucinations and incorrect historical data.
  2. Normative Inconsistency: The selective application of moral judgment, where the AI acts as an inconsistent moral authority.
• Model-Specific Behaviors: It provides a granular comparison of how different model architectures and safety fine-tunings (e.g., Bard vs. ChatGPT) result in different types of errors.

4. Key Results

A. Accuracy and Hallucinations

• Overall Accuracy: Only ~50% of all responses aligned with the human baseline. Accuracy dropped significantly for lower-resource languages (Ukrainian/Russian).
• Model Performance:
  • Google Bard: Showed the highest accuracy in English and Ukrainian but the lowest in Russian (~30% accuracy). It was the most prone to hallucinations, with over 50% of responses containing fabricated details (e.g., false dates, non-existent witnesses).
  • Bing Chat: Performed well in English but struggled with Ukrainian terminology (e.g., "Banderites"), often returning irrelevant or blocked responses.
  • ChatGPT: Showed the most stable performance across languages but frequently produced partially accurate responses (e.g., giving a wide range of victim numbers rather than the specific historical consensus).
• Topic Sensitivity:
  • General Holocaust questions yielded the highest accuracy.
  • Specific episodes (e.g., Lviv pogrom, Luybar massacre) and atrocities against Ukrainians yielded the lowest accuracy.
  • ChatGPT in Ukrainian regarding atrocities against Ukrainians had 0% fully correct responses.

B. Moralization and Normative Framing

• Inconsistency: Moralizing statements were highly inconsistent across languages and models.
  • ChatGPT: Most likely to moralize (>50% of responses), often framing events as "tragedies" and explicitly warning against generalizations or blaming entire nationalities.
  • Bard: Frequently used phrases like "dark chapter" or "bloody violence" and appended moral lessons (e.g., "we must learn from this").
  • Bing Chat: Least likely to moralize (20–25%), usually limiting comments to adjectives like "tragic."
• Linguistic Variance: The same prompt in different languages triggered vastly different moral tones. For example, Bard included moralizing statements in >80% of English prompts about Polish atrocities but only ~20% of Russian prompts.

C. The "Low-Resource" Risk

The study found that users interacting with these models in Ukrainian or Russian are substantially more likely to receive inaccurate, hallucinated, or blocked information compared to English speakers. This creates a "knowledge gap" where the populations most affected by the historical events receive the least reliable information.

5. Significance and Implications

• Threat to Memory Ethics: GenAI risks creating a "non-existent past" that is not only factually wrong but also morally inconsistent. By acting as a pseudo-moral authority without understanding the nuance of contested history, AI can inadvertently validate propaganda or distort the memory of victims.
• Amplification of Distortion: The integration of genAI into social media platforms accelerates the spread of these distortions. Unlike image-generative AI (which tends to homogenize the past), text-generative AI diversifies misrepresentation, creating multiple, conflicting versions of the same historical event depending on the language and model used.
• Policy Recommendations:
  • Refusal Mechanisms: AI developers should implement stricter refusal mechanisms for queries where the model lacks sufficient, verified data, rather than hallucinating an answer.
  • Standardized Moral Framing: There is a need for a "North Star" vision or standard for how AI should normatively frame sensitive historical topics to ensure consistency and avoid misleading moral hierarchies.
  • Contextual Awareness: Future AI development must account for the specific geopolitical and mnemonic contestations of the regions they serve, rather than relying solely on global training data which may be biased toward Global North perspectives.

In conclusion, the paper argues that current genAI applications are ill-equipped to handle high-risk historical memories, posing a significant threat to historical accuracy and the ethical obligations of remembrance, particularly for non-English speaking populations in conflict zones.