Language Models Should be Used to Surface the Unwritten Code of Science and Society

Technical Summary: Language Models Should be Used to Surface the Unwritten Code of Science and Society

Problem Statement
Large Language Models (LLMs) inevitably absorb and reflect human biases embedded in their training data. While existing research focuses on identifying and mitigating these biases, this paper argues for a paradigm shift: leveraging LLM biases as diagnostic tools to uncover and articulate society's "unwritten code." These unwritten codes consist of tacit norms, implicit stereotypes, and subtle conventions (e.g., storytelling preferences in peer review) that invisibly influence societal dynamics but resist formalization because they are rarely explicitly stated by human actors. The core challenge is that current methods often only surface superficial biases (priors) or fail to recover deeper heuristic structures that activate during reasoning processes.

Methodology
The authors propose a conceptual framework to extract and explicitly articulate these unwritten codes through an iterative hypothesis generation process, demonstrated via a case study on the peer review of 26,731 papers across 46 academic conferences.

1. Data Preparation: The study utilizes metadata, peer review scores, and review comments from OpenReview and Paper Copilot. Paper content is represented via extended abstracts (generated by GPT-4o) containing contextual background, key ideas, methodology, and impact, rather than full texts, to ensure attention mechanisms focus on relevant information and mitigate hallucination.
2. Pairwise Construction: A dataset of paper pairs is constructed where one paper received a significantly higher review score than the other (exceeding one standard deviation).
3. Hypothesis Generation Framework:
   • Prior Extraction: The LLM is prompted to generate hypotheses explaining why one paper is stronger than another without specific data input. These represent the model's "normative priors" (e.g., theoretical rigor).
   • Iterative Search (Posterior Update): The model is presented with actual pairwise data. It generates hypotheses to explain observed score differences. For cases where initial hypotheses fail to explain the data ("unexplained" pairs), the model iteratively generates new, distinct hypotheses. This continues until unexplained cases drop below 5%.
   • Self-Consistency Refinement: Generated hypotheses are applied back to the entire dataset. The model evaluates the degree to which each hypothesis explains the score difference, outputting a binary label and a confidence score. A confidence-weighted voting scheme aggregates these judgments to ensure reliability.
4. Human Validation: Human review comments are annotated (using GPT-4o and human experts) to determine if specific aspects (hypotheses) are explicitly mentioned. This allows for a comparison between what humans say (explicit mentions) and what they do (score allocations).

Key Results
The study reveals a distinct divergence between the "priors" (what models and humans claim to value) and the "posteriors" (what actually drives decisions when data is present).

• Divergent Logics: LLMs initially generate hypotheses aligned with normative scientific values (e.g., theoretical rigor). However, as the iterative search progresses to explain difficult cases, the model shifts toward hypotheses centered on storytelling, contextualization, and external connections. These later hypotheses exhibit higher explanatory power and model confidence.
• The Unwritten Code: There is a systematic update from priors to posteriors. Hypotheses related to "internal rigor" decrease in frequency from prior to posterior, while "storytelling" and "interdisciplinary relevance" increase significantly.
• Human-LLM Alignment Gap:
  • Human review comments align with LLM priors (correlation = 0.49), explicitly discussing theoretical rigor.
  • Human comments do not align with LLM posteriors (correlation = –0.14). Despite implicitly rewarding storytelling and contextualization with higher scores, human reviewers rarely articulate these factors in their written feedback.
  • The "unwritten code" (storytelling) accounts for a significant portion of the variance in scores (filling the gap left by explicit criteria) but remains unreported in human feedback.
• Robustness: These patterns hold across different LLM architectures (closed and open-source), out-of-sample data, and alternative paper representations.

Key Contributions

1. Conceptual Framework: The paper introduces a method to "self-reinforce" and amplify biases within LLMs to force the articulation of hidden heuristics. By pushing models to generate self-consistent hypotheses for unexplained decisions, the framework surfaces tacit rules that humans are unable or unwilling to state.
2. Empirical Evidence of Tacit Norms: It provides quantitative evidence that scientific peer review relies heavily on unwritten criteria (storytelling, positioning) that contradict the explicit norms (rigor) cited in reviews.
3. Diagnostic Tool Proposal: The authors propose using LLMs not as automated judges, but as diagnostic tools to surface the gap between explicit articulation and implicit reward, enabling public discussion of revealed values.

Significance and Claims
The paper claims that the primary value of this work is not in automating judgment, but in making the invisible visible. By leveraging the "cognitive shift" from prior to posterior in LLMs, researchers can identify the "hidden curriculum" of societal evaluation systems.

The authors argue that transparency regarding these unwritten codes is essential for equity. Currently, these hidden rules benefit those already familiar with them (e.g., researchers from elite institutions). Making these codes explicit does not merely provide a "cheat sheet" but allows marginalized groups to understand hidden barriers and empowers institutions to reflect on and revise these standards. The paper concludes that implementation should focus on designing systems that log and contrast implicit heuristics against explicit criteria, fostering a more precise and responsible approach to AI in high-stakes evaluation.