💬 NLP

How Do We Engage with Other Disciplines? A Framework to Study Meaningful Interdisciplinary Discourse in Scholarly Publications

This paper proposes a novel framework and a tailored citation purpose taxonomy to quantitatively evaluate how interdisciplinary NLP publications meaningfully engage with and incorporate ideas from other disciplines, addressing the limitations of existing computational metrics.

Original authors: Bagyasree Sudharsan, Alexandria Leto, Maria Leonor Pacheco

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bagyasree Sudharsan, Alexandria Leto, Maria Leonor Pacheco

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a massive library where books from different worlds—like "Computer Science," "Psychology," and "Sociology"—are all mixed together on the same shelves. For a long time, librarians (researchers) have tried to figure out how well these books talk to each other.

The old way of measuring this was like counting how many different colored stickers were on a book's cover. If a book had stickers from five different colors, the librarians assumed it was a deep, rich conversation between those five worlds. But the authors of this paper argue that's misleading. Just because a book mentions a psychology concept in its introduction doesn't mean it actually uses that concept to build its own ideas. It might just be name-dropping.

This paper proposes a new way to measure the conversation: The Citation Engagement Predictor (CEP).

Here is how their new system works, broken down into simple parts:

1. The New "Purpose" Taxonomy (The "Why")

Instead of just counting citations, the authors created a new set of labels to ask: "Why did the author include this specific reference?"

Think of it like analyzing a chef's recipe.

Old Way: "This recipe uses ingredients from Italy, Mexico, and Japan." (Just a list).
New Way: "The chef used the Italian tomato sauce as the foundation for the whole dish," or "The chef used the Mexican spice just to define what 'spicy' means," or "The chef used the Japanese technique to prove their sauce is better."

They created specific categories for these "why" questions, such as:

Basis: The cited idea is the bedrock the whole paper is built on.
Substantiation: The cited idea is used as evidence to prove a claim.
Use: The paper actually uses the cited method or tool directly.
Definition: The paper uses the citation to explain what a word means.
Related Work: The paper just mentions the citation to say, "Hey, other people have done this too," without building on it.

2. The "Location" Clue (The "Where")

The authors also realized that where a citation appears in the paper matters.

If a citation is in the Introduction, it might just be setting the scene (like a background prop).
If a citation is in the Methods or Experiments section, it's likely being used as a tool to do the actual work (like a hammer in a toolbox).

3. The "Citation Engagement Predictor" (The Score)

The authors built a computer model (a "predictor") that combines the Why (Purpose) and the Where (Section) to give a single score from 1 to 5.

Score 1: The paper barely mentions the other work. It's a superficial nod.
Score 5: The paper deeply integrates the other work, using it as a foundation or a core tool.

They tested this by having human experts read 100 pairs of papers and citations to see if the computer's score matched the human feeling of "deep engagement." The computer got pretty good at it, proving that looking at how a citation is used is more important than just counting how many citations there are.

4. What They Found (The Case Study)

They applied this new system to a specific group of papers that sit at the intersection of Natural Language Processing (NLP) (teaching computers to understand human language) and Computational Social Science (CSS) (using computers to study human society).

The Big Discovery:
They found that most of these papers are actually quite shallow in their cross-disciplinary conversations.

The "Name-Dropping" Problem: Many papers cite social science ideas in their introduction just to say, "We know this exists," but then they don't actually use those ideas to build their methods.
The "Deep Dive" is Rare: Only a tiny fraction of papers (about 3%) truly engaged deeply with outside disciplines. These rare papers were the ones that didn't just mention a concept but used it to define their tools or prove their theories.
Internal vs. External: Interestingly, these papers tended to engage deeper with other computer science papers (within their own discipline) than with social science papers (outside their discipline).

The Bottom Line

This paper is like a new magnifying glass for researchers. Instead of just counting how many different colored marbles are in a jar (the old way), this new tool looks at how those marbles are actually being used to build a structure. It reveals that while many researchers say they are doing deep, interdisciplinary work, they are often just skimming the surface. The authors hope this tool will help the scientific community understand the true depth of collaboration between different fields.

Technical Summary: A Framework to Study Meaningful Interdisciplinary Discourse in Scholarly Publications

Problem Statement
Existing computational approaches to measuring interdisciplinarity—such as affiliation diversity, keyword matching, and citation entropy—primarily quantify the quantity and diversity of knowledge sources. However, these metrics fail to capture the depth of engagement, specifically how individual citations are utilized to advance the citing work. Prior taxonomies for citation purpose often lack categories suitable for cross-disciplinary contexts or rely on fixed context windows insufficient for capturing argumentative roles. Consequently, there is a need for a framework that distinguishes between superficial referencing and substantive integration of concepts from outside a paper's primary discipline.

Methodology
The authors propose a framework centered on two core components: a novel citation-purpose taxonomy and a Citation Engagement Predictor (CEP).

Data Collection: The study focuses on the intersection of Natural Language Processing (NLP) and Computational Social Science (CSS). The authors constructed a dataset of 2,031 NLP+CSS articles (published 2015–2025) by training a binary classifier (fine-tuned RoBERTa-base) on abstracts from the ACL Anthology. They extracted 50,050 in-text references, identifying approximately 20,319 as "out-of-discipline" (citing fields outside Computer Science and Linguistics).
Citation Purpose Taxonomy: Through an inductive annotation study, the authors refined a taxonomy to classify the purpose of citations in interdisciplinary work. The categories include:
- Substantiation + Basis: The citation validates a claim and serves as the foundation for a subsequent idea or method.
- Basis: The cited work serves as the explicit foundation for the citing work's method or idea.
- Substantiation: The citation verifies theoretical, empirical, or methodological claims.
- Use: The cited method or tool is used directly with minimal modification.
- Analysis: The cited idea is compared or critiqued to justify an approach.
- Definition: The citation defines a core concept.
- Related Work: The citation provides context or examples without further integration.
- Annotation Process: A dataset of 950 in-text citations was manually annotated by authors and graduate students, achieving a Krippendorff's $\alpha$ of 0.596.
Automatic Classification: The authors evaluated three models for predicting citation purpose: fine-tuned RoBERTa, Llama3-8b, and Qwen3-8b. Qwen3-8b achieved the best performance (Weighted F1: 0.60).
Citation Engagement Predictor (CEP): To measure engagement depth, the authors developed an ordinal logistic regression model. This model predicts a 5-point engagement score (1 = low, 5 = high) for a publication's engagement with a specific cited work. The model utilizes features derived from the frequency of citations appearing in specific sections (Introduction, Method, Experiments, Conclusion, Appendix) and with specific purposes. The "Related Work" category and baseline sections were dropped to avoid linear dependency.

Key Contributions

Novel Annotation Framework: A taxonomy specifically designed to capture citation purposes in interdisciplinary NLP work, distinguishing between surface-level referencing and deep integration (e.g., "Substantiation + Basis").
Dataset and Classifier: A dataset of 950 annotated in-text citations from NLP+CSS papers and a state-of-the-art classifier (Qwen3) for automatically predicting citation purpose.
Citation Engagement Predictor (CEP): A statistical model that synthesizes signals from citation purpose and section location to generate a single, interpretable measure of engagement depth.
Empirical Analysis: A large-scale case study analyzing 34,786 article-citation pairs in NLP+CSS to evaluate the utility of the framework.

Results

Model Performance: The automatic classifier (Qwen3) achieved a Macro F1 of 0.56. The CEP model achieved a Mean Absolute Error (MAE) of 0.484 and a Quadratic Weighted Kappa (QWK) of 0.794 on a held-out set of 100 annotated article-citation pairs.
Feature Importance: Regression coefficients indicate that the purpose of a citation is a stronger predictor of engagement depth than the section in which it appears. Specifically, citations labeled "Basis" and those appearing in the "Appendix" or "Method" sections correlate with higher engagement scores.
Engagement Distribution: In the NLP+CSS dataset, the majority of citations (55.8%) received a Level 1 (low) engagement score. Only 2.9% of citations achieved Level 4 or 5 (high engagement).
Interdisciplinary vs. Within-Discipline: The analysis reveals that authors tend to engage more deeply with within-discipline citations than interdisciplinary ones. Interdisciplinary citations are more frequently associated with "Substantiation" and "Definition" purposes, whereas within-discipline citations often involve methodological "Use."
Qualitative Findings: High-engagement interdisciplinary papers (e.g., those integrating cognitive science) tend to weave cited concepts throughout multiple sections and purposes, whereas lower-engagement papers often cite interdisciplinary work only for background or definition without building upon it.

Significance and Claims
The paper claims that its framework provides a necessary complement to existing interdisciplinarity metrics by shifting the focus from what is cited to how it is cited. By operationalizing the depth of engagement, the framework allows researchers to identify whether interdisciplinary work is merely "name-dropping" concepts from other fields or substantively integrating them into methodological and theoretical choices. The authors position this work as a "first step" toward a broader framework for studying interdisciplinary engagement across fields, specifically noting that their findings are grounded in the NLP+CSS context and that validation in other domains remains future work. They explicitly caution that their automated approach carries uncertainty due to parsing errors and the reliance on automatic classifiers for the large-scale analysis.