Attributing Culture-Conditioned Generations to Pretraining Corpora

1. Problem Statement

Large Language Models (LLMs) exhibit significant cultural biases in open-ended generative tasks (e.g., narrative writing, dialogue). Specifically, they demonstrate limited knowledge and generate templated, generic outputs for less prevalent cultures, while over-representing high-frequency cultures. While previous work has established that these biases stem from uneven representation in pretraining corpora, there is a lack of granular understanding regarding how specific cultural associations in model outputs are formed.

The core problem addressed is the inability to distinguish whether a model's generation of a cultural symbol (e.g., "kimono" for Japan) is due to:

1. Memorization: Direct recall of culture-symbol pairs from the pretraining data.
2. Generalization: Inference based on weak associations or diffusion from high-frequency terms.
3. Bias: The model favoring high-frequency tokens regardless of the specific cultural prompt.

2. Methodology: The MEMOED Framework

The authors propose MEMOED (MEMOrization from pretraining document), a framework designed to attribute culture-conditioned generations back to specific patterns in the pretraining data. The study utilizes OLMo-7B and its pretraining corpus, Dolma, which are open-source and indexed, allowing for document-level tracing.

A. Data Collection

• Scope: 110 cultures across food and clothing topics.
• Generation: The model was prompted to describe food/clothing for a specific culture (e.g., "My neighbor is [Culture]. He probably likes to eat...").
• Extraction: 100 generations per gender setting (300 total per culture) were processed using LLAMA-3-70b to extract "symbols" (entities like specific dishes or garments).

B. Classification of Associations

The framework categorizes culture-symbol associations into four types:

1. Memorized Association: The symbol and culture co-occur frequently in the pretraining data within a meaningful context.
2. Diffuse Association: Symbols (e.g., "rice," "t-shirt") generated for >50% of cultures without specific grounding in any single culture's pretraining documents.
3. Cross-Culture Generalization: A symbol memorized for Culture A is generated for Culture B (e.g., "kimono" generated for Korea).
4. Weak Association Generalization: Broader concepts inferred from memorized symbols (e.g., "robe" inferred from "kimono").

C. The MEMOED Pipeline (Technical Core)

To determine if a symbol is Memorized, MEMOED analyzes the pretraining documents:

1. Relevance Filtering: A document $D$ is considered contributory if the culture $C$ and symbol $S$ appear within a specific distance and the document is topically relevant to $C$.
   • Distance Metrics: Minimum token distance ($d_{TOK}$) and minimum sentence distance ($d_{SENT}$).
   • Context Relevance: Measured via Document-Signal to Noise Ratio ($d_{SNR}$). This calculates the log-ratio of the frequency of the target culture $C$ versus all other cultures in the document. High $d_{SNR}$ indicates the document is specifically about $C$.
2. Contribution Score (CS): The ratio of contributory documents to the total number of documents where the symbol appears.
3. Z-Score Thresholding: The CS is calculated for all cultures generating a specific symbol. If a culture's CS is an outlier (Z-score $\ge$ 2.6), the symbol is classified as Memorized for that culture.

D. Analyzing Non-Memorized Associations

• Diffuse Association: Analyzed via "Overshadowing Ratio," comparing the frequency of diffuse symbols against memorized symbols in the corpus.
• Cross-Culture Generalization: Investigated using Topic Modeling (LDA) on documents where the source culture, target culture, and symbol co-occur to find shared thematic keywords (e.g., "religion" linking Hijab, Iran, and Saudi Arabia).
• Weak Association: Validated by prompting the model to define memorized symbols and checking if the generated "weak" symbols match the broader definitions.

3. Key Results

A. Memorization is Frequency-Dependent

• There is a strong positive correlation between the number of pretraining documents for a culture and the number of memorized symbols generated for it (Food: Spearman $\rho=0.67$; Clothing: $\rho=0.54$).
• Long-tail cultures (low-frequency in pretraining) suffer from "periodic forgetting" during training, resulting in almost zero memorized symbols. For clothing, ~60% of cultures had no memorized symbols.

B. The Nature of Memorized Symbols

• Not Just Emblematic: MEMOED identifies memorized symbols that are not culturally emblematic but are simply frequent in the data (e.g., "business suit" for Japan, "western bridal gown" for India).
• Human Alignment: The framework's classification aligns well with human annotators (Weighted F1: 0.845 for clothing, 0.670 for food).

C. Diffuse Association is Frequency-Driven

• Diffuse symbols (e.g., "chicken," "rice") appear in the pretraining data 1000x more frequently than memorized symbols.
• This massive frequency gap causes the model to "overshadow" specific cultural knowledge with generic, high-frequency tokens during sampling.

D. Cross-Culture Generalization Patterns

• High-to-Low Flow: Cultures with high pretraining frequency are more likely to have their memorized symbols generated for other cultures. Conversely, low-frequency cultures are more likely to receive symbols from high-frequency cultures.
• Thematic Co-occurrence: Generalization often occurs due to shared topics in pretraining (e.g., "Hijab" appearing in documents discussing both Iran and Saudi Arabia under the topic of "Islamic politics").

E. Distribution of Associations

• Diffuse Association Dominance: While diffuse symbols make up a small fraction of unique symbols, they constitute 79.2% (food) and 91.12% (clothing) of total generated responses.
• Memorization Scarcity: Memorized symbols average only 4.12% of food responses and 0.76% of clothing responses.
• Variance: High-frequency cultures (e.g., Mexico) show high memorization rates (~42%), while others (e.g., Trinidad) show 0%.

4. Key Contributions

1. MEMOED Framework: A novel, reproducible method to attribute LLM generations to specific pretraining documents, distinguishing between memorization and generalization.
2. Granular Taxonomy: The identification of four distinct types of culture-symbol associations, moving beyond a binary "bias vs. no bias" view.
3. Empirical Evidence of Imbalance: Quantitative proof that LLMs rely heavily on high-frequency, diffuse symbols for low-frequency cultures, leading to a "knowledge collapse" where specific cultural details are lost.
4. Correlation Analysis: Establishing strong statistical links between pretraining data frequency and the model's ability to memorize vs. generalize cultural knowledge.

5. Significance and Implications

• Diagnosis of Bias: The paper provides a diagnostic tool to understand why a model fails on specific cultures (lack of memorization due to data scarcity) versus why it hallucinates generic content (overshadowing by high-frequency tokens).
• Mitigation Strategies: The findings suggest that simply fine-tuning may not be enough; effective mitigation requires data augmentation for long-tail cultures or unlearning of high-frequency diffuse associations to force the model to retrieve specific cultural knowledge.
• Future Research: The framework sets a precedent for "data attribution" in LLMs, encouraging research into linking model behavior directly to pretraining corpus statistics rather than just output metrics.

Limitations: The study is currently constrained to OLMo-7B due to the requirement for searchable, open pretraining data. However, the methodology is theoretically transferable to other models with accessible training corpora.