Scaling In, Not Up? Testing Thick Citation Context Analysis with GPT-5 and Fragile Prompts

1. Problem Statement

The paper addresses a fundamental tension in Citation Context Analysis (CCA), a method used to interpret the function of in-text references (e.g., whether a citation is supportive, critical, or perfunctory).

• The "Scaling-Up" Limitation: Traditional CCA attempts to scale by creating automated classifiers that assign stable, typological labels to citations across large corpora. However, scholars like Gilbert (1977) argue this approach is sociologically fragile because citation meaning is often ambiguous, context-dependent, and plural. Different readers may legitimately interpret the same citation differently based on standpoint and domain knowledge.
• The "Scaling-In" Gap: While Large Language Models (LLMs) have renewed the ambition to automate classification, they have rarely been tested on their ability to handle interpretative plurality. The core problem is whether LLMs can be used not to force a single "correct" label, but to "scale in"—deepening the analysis of a single hard case by generating a structured space of plausible, text-grounded alternative interpretations.
• The Specific Case: The study focuses on Footnote 6 in Chubin and Moitra (1975), a canonical "hard case" where a citation appears routine but contains subtle, potentially critical undertones regarding the distinction between "references" and "citations."

2. Methodology

The author employs a two-stage prompting pipeline using OpenAI's GPT-5 (in a high-reasoning configuration) as a "guided co-analyst." The study utilizes a 2×3 experimental design to test prompt sensitivity.

A. The Two-Stage Pipeline

1. Stage 1 (Surface Classification):
   • Input: Footnote 6 text only.
   • Task: Classify the citation into Chubin and Moitra's (1975) hierarchical typology (e.g., Essential, Supplementary, Negational) and formulate expectations about the cited works based strictly on the surface text.
   • Goal: Establish a stable baseline similar to traditional "scaling-up" automation.
2. Stage 2 (Interpretative Reconstruction):
   • Input: Stage 1 outputs + Full texts of the citing paper (Chubin & Moitra, 1975) and cited sources (Price, 1970; Gilbert & Woolgar, 1974).
   • Task: Move beyond surface labels to generate five distinct hypothesis–justification pairs. These must complicate the surface reading, be grounded in textual cues, and engage with cross-document evidence.

B. Experimental Design (2×3)

The Stage 2 prompt was varied across six configurations to test how scaffolding and framing influence outputs:

• Base Prompt Structure (2 levels):
  • 4-step: Explicit scaffolding requiring the model to perform expectation checks, cue analysis, extended-context analysis, and then hypothesis generation.
  • 1-step: Direct instruction to generate five hypotheses without intermediate steps.
• Nudging Conditions (3 levels):
  • No-nudge: No examples provided.
  • Toward: Examples provided that suggest plausible interpretations for this specific case (e.g., lineage management, muted critique, audience bridging).
  • Away: Examples provided suggesting interpretations less plausible for this case (e.g., didactic signposting, praise-as-ridicule) to stress-test the model's adherence to framing.

C. Data Analysis

• Data: 90 runs total, generating 450 distinct hypotheses.
• Coding: Inductive close reading identified 21 recurring binary codes (subthemes) representing interpretative moves (e.g., "Preempt," "Canon," "MuteGW," "Bridge").
• Statistical Modeling: Linear Probability Models (LPMs) were used to estimate how prompt variations (scaffolding and nudging) shift the probability of specific codes appearing. Average Marginal Effects (AMEs) were calculated to measure the impact of design choices.

3. Key Contributions

1. Methodological Shift (Scaling In): Proposes a framework for using LLMs not to automate classification convergence, but to expand the interpretative space of a single case, treating disagreement and ambiguity as the object of study rather than noise.
2. Prompt Sensitivity Analysis: Demonstrates that prompt design (scaffolding and examples) acts as a methodological lever that systematically redistributes interpretative attention and vocabulary, rather than just retrieving a static "truth."
3. Guided Co-Analyst Framework: Introduces a reproducible, two-stage pipeline that separates surface classification from deep reconstruction, allowing for the inspection of how models construct arguments.
4. Empirical Validation of "Thick Description": Shows that LLMs can perform "thick description" by cross-referencing full texts, detecting mismatches in citation practices, and generating text-grounded hypotheses that rival human expert analysis.

4. Key Results

A. Surface Stability (RQ1)

• GPT-5's Stage 1 classification was highly stable, consistently labeling Footnote 6 as "Supplementary" (either Perfunctory or Additional Information). This confirms that under "scaling-up" constraints, the model behaves like a traditional classifier.

B. Interpretative Plurality (RQ2)

• In Stage 2, the model generated a structured space of 450 hypotheses covering 21 distinct subthemes.
• The model successfully identified textual hinges (e.g., the word "reiterated," the lack of citation to Price in Gilbert & Woolgar's definition) and used them to construct diverse arguments ranging from "preempting critique" to "genealogical rewriting."
• The outputs were not random; they were textually anchored and often combined multiple interpretative moves (e.g., Preempt + Agile + Agency).

C. Prompt Sensitivity (RQ3)

• Scaffolding (4-step vs. 1-step): The 4-step prompt increased the frequency of genealogical and authority-based readings (e.g., Canon, UseGW) and decreased pragmatic/flexible readings (Pragma, Agile). The explicit steps forced the model to treat specific phrasings as consequential.
• Nudging (Toward/Away):
  • Toward significantly increased readings related to lineage, priority, and muted critique (PrioP, MuteGW, Bridge), aligning with the examples provided.
  • Away increased didactic (Teach) and social/audience (SSS) readings, even when these were less plausible for the specific case.
• Lexical Echo: The model incorporated vocabulary from the "nudge" paragraphs (e.g., "genealogy," "slippage") into its hypotheses, proving that examples seed not just storylines but the rhetorical repertoire used to articulate them.

D. Comparison with Gilbert (RQ4)

• Cue Detection: GPT-5 successfully detected the same textual hinges Gilbert (1977) used (specifically the "introduced... then reiterated" phrasing and the missing citation to Price).
• Resolution Divergence: While Gilbert resolved these cues into a veiled admonishment (criticism of Gilbert & Woolgar for failing to credit Price), GPT-5 almost exclusively resolved them into lineage management and positioning (normalizing the distinction, muting conflict, or aligning with bibliometric authority).
• Reasoning: The model avoided the "admonishment" reading, likely due to safety alignments favoring cooperative/non-escalatory continuations, or because the prompt did not explicitly license a "reprimand" interpretation.

5. Significance and Implications

• Redefining Automation: The paper argues that the value of LLMs in interpretative sociology is not faster labeling, but assisting in the production of alternative readings. It shifts the human role from "coder" to "curator" of a model-generated interpretative space.
• Prompt Engineering as Methodology: The study demonstrates that prompt design is not a neutral technical step but a core methodological variable. Researchers must explicitly document and vary prompts to understand how they shape the "interpretative space."
• Limitations of Current Models: The failure of GPT-5 to reproduce Gilbert's "admonishment" reading highlights that current models have alignment biases (e.g., towards agreeableness) that can suppress conflict-oriented or critical interpretations unless explicitly prompted.
• Future Directions: The "scaling-in" approach offers a path for inspectable and contestable CCA. By making the interpretative steps explicit (via prompt chaining), the workflow becomes reproducible and open to critique, bridging the gap between computational efficiency and hermeneutic depth.

In conclusion, the paper successfully demonstrates that LLMs can function as guided co-analysts capable of generating "thick descriptions" of citation contexts. However, this capability is highly sensitive to prompt framing, requiring researchers to carefully design prompts that balance structural guidance with the openness needed to capture interpretative plurality.