💻 computer science

The Reciprocal Impact of Science and Software: A Cross-Corpus Analysis of How Research Shapes Software and Software Enables Research

This study constructs a large-scale cross-corpus graph linking scientific literature and software repositories to reveal that while science and software co-evolve through distinct, complementary layers of influence, current measurement methods remain sparse and sensitive to methodological choices, preventing definitive conclusions about their reciprocal impact.

Original authors: Audris Mockus

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Audris Mockus

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of scientific discovery as a massive, bustling city. For a long time, this city has been split into two separate districts that rarely talk to each other:

The Library District: This is where scientists publish their papers. It's measured by how many people borrow books (citations) and how famous the authors are.
The Workshop District: This is where the actual tools, code, and software are built. It's measured by how many people "star" a project on GitHub or how many forks it has.

The problem? These two districts are using different maps. A scientist might build a revolutionary tool in the Workshop, but the Library doesn't know about it. Conversely, a famous paper in the Library might rely on a tiny, invisible tool in the Workshop that no one ever mentions.

This paper, by Audris Mockus, tries to build a bridge between these two districts to create a single, unified map of how science actually works. The author calls this the "Science-Software Supply Chain."

Here is what the study found, explained through simple analogies:

1. The Two-Way Street

The researchers connected two giant databases: one containing almost all public software code (World of Code) and one containing scientific papers (Semantic Scholar/OpenAlex). They looked at the connections in two directions:

Direction A (Science $\rightarrow$ Software): Which scientific papers actually shape the tools scientists use?
- The Surprise: It's not the most famous papers with the most citations. Instead, the tools that get adopted the most are the ones that help scientists package and organize their work (like "nf-core" or "Nextflow").
- The Analogy: Think of it like a construction site. The most famous architect (the highly cited paper) might design a beautiful building, but the tools that get used by every construction crew are the scaffolding and the cranes (the packaging tools). The paper counts the architect's fame, but the software counts the scaffolding's utility.
Direction B (Software $\rightarrow$ Science): Which software tools actually enable new scientific discoveries?
- The Surprise: The tools that enable the most science are often invisible. They are the "hidden infrastructure" like PyTorch (for AI) or data visualization tools. They are rarely mentioned in paper titles, but thousands of other projects depend on them.
- The Analogy: Imagine a massive city where everyone drives on roads. You don't see the roads in the news reports about the cars; you only see the cars. But if the roads disappeared, the cars would stop. These software tools are the roads. They are essential, but because they are everywhere, no one stops to write a paper saying, "I used a road today."

2. The "Star" Rating Trap

In the software world, people often judge a tool's importance by how many "stars" (likes) it gets on GitHub. The study found that stars are a terrible predictor of real usage.

The Analogy: Imagine a restaurant. A place with 500 "likes" on a social media app might be a trendy spot that people visit once for a photo. But a place with only 10 likes might be the local bakery that supplies bread to 10,000 other restaurants every morning.
The Finding: The study found a very weak connection between "stars" and "dependencies" (how many other projects actually use the code). A tool with 150 stars might be used by 9,000 other projects, while a tool with 50,000 stars might be used by very few. Popularity does not equal utility.

3. The "Invisible" Gap

The researchers tried to count how often papers mention software. They found a massive gap.

The Problem: When scientists write papers, they often mention software in the middle of the text (the "methods" section) rather than in the formal reference list. Automated tools often miss these mentions.
The Result: The link between a paper and the software it uses is "sparse" and "noisy." It's like trying to map a family tree using only the names people shout at a party, rather than looking at the actual birth certificates. Because the data is so incomplete, the researchers couldn't give a single, perfect number for how much software impacts science. Instead, they showed that different ways of measuring give different answers, and we need to be careful not to trust just one number.

4. The New Workers: AI Agents

The study also looked at a new phenomenon: AI coding agents (robots that write code).

The Finding: These AI agents are starting to appear in scientific software projects. They aren't replacing humans yet, but they are joining the workforce.
Who uses them? It's not about the type of science (like biology vs. physics). Instead, AI agents are used by fast-moving, active projects. If a project is young and its developers are coding furiously, they are more likely to use AI helpers. It's like a startup moving fast and using new gadgets, rather than a slow, established factory.

The Big Takeaway

This paper argues that we need to stop looking at science and software as separate things.

Old View: Science is papers; Software is just a tool.
New View: Science is a supply chain. Papers and software are co-equal goods.

To truly understand scientific progress, we can't just count citations or "stars." We need to see the dependencies—the invisible web of code that holds everything together. Just as a city needs to know about its water pipes and power grids, not just the famous buildings, the scientific community needs to value the "hidden hero" software that makes discovery possible, even if no one writes a paper about it.

In short: The most important tools in science are often the ones you never hear about, and the most popular tools are often the ones you don't actually need. To fix this, we need a better map that connects the Library to the Workshop.

Technical Summary: The Reciprocal Impact of Science and Software

Problem Statement
Scientific knowledge and software co-evolve, yet they are recorded in disjoint ledgers: software in version-control ecosystems (e.g., GitHub) and science in bibliographic databases (e.g., Semantic Scholar, OpenAlex). This separation renders the "Science-Software Supply Chain" (S3C) unobservable. Current metrics fail to capture the reciprocal relationship: bibliometrics count citations but miss software reuse, while software metrics (stars, forks) do not reflect scholarly impact. Furthermore, existing studies on software mentions in literature suffer from low recall and unreliable linkage to actual artifacts, while dependency analyses are often confined to single ecosystems without resolving developer identities. The paper posits that without a global, identity-resolved link between papers and software artifacts, the provenance of scientific production remains opaque, hindering risk management (e.g., identifying "hidden hero" dependencies) and efficient credit allocation.

Methodology
The authors construct a typed cross-corpus graph linking the near-complete public version-control history (World of Code, V2604) with bibliographic corpora (Semantic Scholar and OpenAlex).

Data Sources:
- Software: World of Code (WoC) provides commits, authors, projects, and project-to-project dependency edges.
- Science: Semantic Scholar and OpenAlex provide papers, citations, and software mentions.
- Seed: 18,247 science repositories from the SciCat dataset, curated via LLM classification for scientific intent.
Graph Construction:
- The graph contains 69.8 million edges across eight layers, including depends on (software-to-software), mentions doi (software citing papers), mentions repo (papers naming software), and identity bridges (same as) reconciling authors and DOIs.
- Identity resolution utilizes the WoC author-aliasing map and ORCID/affiliation data to link developers to canonical identities.
Impact Measures (Reciprocal Questions):
- RQ1 (Science $\to$ Software): Measures the "reach" of papers into software. For a paper, this is the count of distinct science repositories citing it. For an author, it is the reach of repositories via their papers.
- RQ2 (Software $\to$ Science): Measures the "enabling" of science by software.
  - Strict Measure: Papers naming a repository (sparse, $n=137$ ).
  - Proxy Measure: Ecosystem Impact, defined as the reverse depends on in-degree (number of distinct projects depending on a repository). This is used as a proxy for scientific enabling because the direct paper-to-software channel is too sparse to rank effectively.
Validation:
- Internal: Comparison of two independent extractors (Semantic Scholar full-text vs. OpenAlex abstracts) to assess recall.
- External: A held-out Softcite v2 gold corpus (human-curated biomedical/economics mentions) used to bound recall and identify structural biases (e.g., methods-section mentions vs. abstracts).
- Triangulation: Analysis performed across multiple anchors (SciCat, JOSS, Softcite tool names, CRAN/conda registries) to test robustness.

Key Results

Distinct Ecosystem Strata: The two directions of impact surface largely disjoint layers of the ecosystem.
- Science $\to$ Software: Dominated by a reproducibility/packaging layer (e.g., nf-core, Nextflow, Singularity, Bioconda) and sequence-analysis tools (SAMtools, STAR). These tools are adopted wholesale by many pipelines.
- Software $\to$ Science (Proxy): Dominated by an invisible ML/data-science infrastructure tier (e.g., PyTorch, seaborn, NLTK, einops). These tools have high dependency in-degree but are rarely cited directly as the enabling substrate.
- Scholars: The authors underpinning the ecosystem are primarily the toolmakers (e.g., Paolo Di Tommaso for Nextflow, Heng Li for SAMtools).
Weak Coupling of Metrics:
- Reuse vs. Popularity: Dependency in-degree is only weakly correlated with GitHub stars (Spearman $\rho = 0.36$ ). High-dependency tools like cloudflare/circl (9,459 dependents) may have very few stars (150).
- Reuse vs. Citation: The correlation between software reuse and scholarly citation is fragile and gap-sensitive.
  - When pairing via papers that name the repository ( $n=137$ ), the correlation is undetectable ( $\rho = 0.05$ , CI straddling zero).
  - When pairing via DOIs declared by the repository ( $n=1,067$ ), a weak positive correlation emerges ( $\rho = 0.13$ , CI [0.07, 0.19]).
  - The authors conclude that the sign of the correlation is an artifact of which linkage gap is tolerated; strong decoupling claims are premature given current data density.
Linkage Limitations:
- Recall is Low: The external gold benchmark reveals that the graph's recall for methods-section mentions is near zero. The graph captures a "thin, structurally-biased slice" (abstracts, landing pages, references) rather than the full methods text.
- Seed Bias: The SciCat seed covers only ~11% of curator-annotated tools in WoC and overlaps poorly with JOSS repositories. Triangulating with multiple anchors confirms that no single seed captures the full landscape.
AI Coding Agents:
- The study identifies 58 science repositories adopting AI coding agents (2023+).
- Adoption is associated with development intensity (active months, commit volume) and relative youth, but not with field, popularity (stars), or team size.
- Post-adoption outcome studies are deferred due to insufficient time for impact stocks (citations/dependencies) to accumulate.

Significance and Claims
The paper frames its contribution as feasibility evidence rather than a definitive accounting.

Conceptual Shift: It successfully reframes scientific production as a supply chain where papers and software are co-equal goods, demonstrating that a global, identity-resolved "bill of materials" is constructible.
Methodological Insight: It proves that reading a single global graph in different directions reveals distinct, weakly coupled strata of impact. Crucially, it demonstrates that citation counts and popularity (stars) cannot substitute for dependency-based reuse as a measure of scientific impact.
Caveats: The authors are explicit that the quantitative coupling between reuse and citation is currently too sensitive to linkage gaps (selection bias, recall issues) to support strong causal claims. The "sign" of the correlation flips depending on the pairing method.
Future Utility: The constructed graph provides the necessary substrate to track emerging phenomena (like AI agents) and to eventually manage supply-chain risks (e.g., identifying unmaintained "hidden hero" libraries) that are currently invisible to bibliometrics.

The paper concludes that while the current linkage density is insufficient for precise point estimates, the directionality of the findings is robust enough to motivate the inclusion of software-aware, dependency-grounded metrics in the science-of-science toolkit.