Research Borderlands: Analysing Writing Across Research Cultures

Technical Summary: Borderlands: Analysing Writing Across Research Cultures

Problem Statement
While improving the cultural competence of language technologies is a recognized priority, recent works often fail to engage with the specific communities they study. Instead, they rely on synthetic setups and imperfect proxies for culture, such as nationality or language, which lack grounding in specific tasks or cultural contexts. Consequently, the concept of "culture" in Large Language Model (LLM) evaluation remains vaguely defined. There is a critical need to understand how writing norms vary across specific research communities and to evaluate whether LLMs can effectively adapt to these nuanced, often undocumented, cultural expectations.

Methodology
The authors adopt a human-centered, bottom-up approach to define and measure cultural norms within research communities, focusing specifically on the task of adapting research paper introductions across different scientific fields.

1. Data Collection via Mixed Methods:

   • Survey (N=78): A preliminary survey confirmed that interdisciplinary researchers frequently adapt papers for different communities (e.g., after rejection or to fit a new venue), with the introduction section being the most commonly adapted component.
   • Interviews (N=10): Semi-structured interviews were conducted with senior interdisciplinary researchers who act as experts in navigating between communities. Participants provided examples of their own papers rewritten for different venues and explained their adaptation strategies.

2. Framework Development:
   Through qualitative analysis of interview transcripts, the authors developed a framework of four categories of cultural norms:

   • Structural Norms: Length, and the presence of artifacts (tables/figures).
   • Stylistic Norms: Jargon/specialized language, readability, formality, and verbosity.
   • Rhetorical Norms: The use of quantitative evidence, figurative language, framing (highlighting specific values), and narrative organization (the ordering of background, objectives, methods, and results).
   • Citational Norms: Canonicity (citing foundational works) and engagement style (e.g., use of direct quotes).

3. Operationalization (Evaluation Suite):
   The authors translated this framework into a suite of computational metrics:

   • Length/Artifacts: Word counts, sentence counts, and regex-based detection of table/figure mentions.
   • Jargon: Specificity scores using Normalized Pointwise Mutual Information (NPMI).
   • Formality: Sentence-level scores using a DeBERTa-large model fine-tuned on GYAFC.
   • Readability: Average Flesch reading-ease scores.
   • Quantitative Evidence: An LLM-as-a-judge setup (Llama 3.1 70B) to detect numerical evidence.
   • Narrative Organization: A multinomial classifier (SCIM) to categorize sentence functions and measure their positional skew.
   • Framing: A lexicon-based classifier to detect encoded research values (e.g., efficiency, fairness, novelty).

4. Quantitative Analysis:

   • Human Writing Analysis: The suite was applied to a corpus of 81,178 research paper introductions from 11 computer science communities (e.g., NLP, ML, HCI, Education) to surface latent variations.
   • LLM Evaluation: The authors evaluated the cultural competence of five LLMs (GPT-3.5, GPT-4o Mini, Llama 3.1/3.3, Mistral) by prompting them to adapt introductions from source communities to target communities. The changes in metric values were compared against the actual distributions of human-written papers in the target communities.

Key Results

• Human Writing Variations: The computational metrics successfully recovered distinct cultural norms across the 11 communities at scale. For instance, Education papers showed high variance in readability, while ML and NLP communities exhibited strong, consistent norms regarding the quantity of quantitative evidence. Specificity scores (jargon) varied significantly, with Education being the most distinctive community.
• LLM Cultural Competence:
  • Vocabulary Adaptation: LLMs successfully adapted vocabulary, increasing specificity scores to match target communities.
  • Homogenization: Across all other metrics, LLMs tended to move features in a single direction regardless of the target community. For example, models consistently shortened text, reduced the mention of tables and figures, lowered readability, and increased the density of quantitative evidence.
  • Failure to Adapt: Models only "succeeded" in adapting when their default homogenizing trend happened to align with the target community's norms (e.g., shortening text for NLP, where introductions are naturally shorter). In cases where the target community required longer text or different structural arrangements (e.g., Education), the models failed to adapt, instead imposing a uniform style.

Significance and Claims
The paper claims to illustrate the efficacy of a human-centered approach to measuring cultural norms in both human-written and LLM-generated texts. By centering community members and focusing on a real-world task (writing adaptation), the authors provide a more ecologically valid alternative to top-down approaches that rely on broad proxies like nationality.

The work highlights a significant limitation in current general-purpose LLMs: while they can adjust vocabulary, they struggle to adhere to the structural, stylistic, and rhetorical nuances of specific research cultures, tending instead to homogenize scientific writing. The authors argue that this homogenization poses a risk to the diversity of ideas and writing styles in the scientific community. They conclude that future scientific writing tools should likely focus on interactive assistance that exposes community barriers rather than fully automating the writing process, preserving the human role in shaping community values through writing. The authors have open-sourced their evaluation suite to support future research in the science of science and the evaluation of cultural competence.