Comparing the outputs of intramural and extramural grants funded by National Institutes of Health

This study compares NIH intramural and extramural grants from 2009 to 2019, revealing that while extramural funding is more cost-effective for generating academic publications and citations, intramural funding is more efficient at producing research that directly influences future clinical work.

The Gist

Imagine the National Institutes of Health (NIH) as a massive, wealthy parent trying to solve the world's health problems. This parent has two distinct ways of spending their money to get things done:

1. The "In-House Team" (Intramural): They hire their own scientists to work in their own government laboratories. These scientists are on the payroll, don't have to write grant applications, and focus entirely on the parent's specific goals.
2. The "Outsourced Contractors" (Extramural): They give grants (money) to thousands of universities and hospitals across the country. These scientists have to compete for the money, write proposals, and often have to find extra funding from their own universities to keep their labs running.

This paper asks a simple but huge question: Which way of spending money gets better results?

To find out, the researchers looked at nearly 100,000 projects funded between 2009 and 2019. They compared the "In-House Team" against the "Outsourced Contractors" using a few different yardsticks.

The Analogy: The Restaurant vs. The Food Truck

Think of the In-House Team as a high-end, private restaurant owned by the parent.

• Pros: The owner (NIH) can tell the chefs exactly what to cook. They don't have to worry about rent or marketing. They can focus on a very specific, difficult dish (like a complex clinical cure) without distraction.
• Cons: It's expensive to run. The owner pays for everything (ingredients, staff, electricity) with no help from outside.

Think of the Outsourced Contractors as thousands of food trucks in a city.

• Pros: They are incredibly efficient at churning out popular, tasty food (scientific papers). Because they are in a competitive market, they produce a huge volume of work. The city (universities) often chips in to help pay for the trucks' maintenance (indirect costs).
• Cons: They are chasing trends to stay in business. They might spend a lot of time writing menus (grant applications) instead of cooking.

What Did They Find?

The researchers discovered that neither side is "better" overall; they are just better at different things.

1. The "Volume and Fame" Game (Academic Metrics)

If you measure success by how many papers are published and how often they are cited by other scientists (the standard way universities judge professors), the Outsourced Contractors (Universities) win.

• The Metaphor: The food trucks are churning out delicious burgers and tacos faster and cheaper per dollar. For every $1 the NIH spends, the universities produce more "academic clout" (papers and citations).
• Why? Universities have a culture that rewards publishing. Also, because universities often share the cost (paying for student labor, lab space, etc.), the NIH gets more "bang for their buck" in terms of raw output.

2. The "Real-World Impact" Game (Clinical Metrics)

If you measure success by how much the research actually helps doctors treat patients (clinical citations), the In-House Team (NIH Labs) wins.

• The Metaphor: The private restaurant is better at cooking the specific, difficult, life-saving meal that the owner ordered. Even though it costs more per plate, the food is exactly what the family needed to get healthy.
• Why? The NIH scientists don't have to worry about writing grant applications or teaching classes. They can focus entirely on the NIH's mission: improving human health. Their research is more likely to bridge the gap between "the lab bench" and "the hospital bed."

The "Hidden Costs" Twist

The researchers also realized that the "food trucks" (universities) aren't paying the full price. The NIH pays the grant, but the universities often subsidize the rest (like student salaries and overhead) without getting reimbursed.

• Even when the researchers tried to account for these hidden costs (pretending the universities paid a bit more), the pattern held true: Universities are still cheaper for making papers; NIH labs are still better at making clinical breakthroughs.

The Big Takeaway

The paper concludes that the NIH shouldn't try to make one system do everything. Instead, they should play to their strengths:

• Use Universities (Extramural) for generating a massive amount of new knowledge, training the next generation of scientists, and keeping the scientific community buzzing with new ideas.
• Use NIH Labs (Intramural) for the heavy lifting: tackling the hardest, most specific health problems that require long-term focus and direct alignment with the agency's mission, especially when it comes to getting treatments to patients.

In short: If you want a lot of new ideas and publications, hire the contractors. If you want a specific, life-saving cure developed without distraction, hire your own team. The smartest strategy is to use both.

Technical Summary

1. Problem Statement

The National Institutes of Health (NIH) utilizes two distinct funding mechanisms:

• Intramural Research: Conducted within NIH-owned laboratories by government-employed scientists.
• Extramural Research: Conducted at external institutions (universities, medical schools) via competitive grants.

While the NIH spends billions on both (approx. $5B intramural vs. $39B extramural in FY2022), there is a lack of empirical evidence comparing their cost-effectiveness and scientific output profiles. The study addresses the question: Do these two mechanisms produce different types of research value, and which is more efficient for specific outcomes (e.g., academic publications vs. clinical translation)?

2. Methodology

Data Collection and Sample:

• Source: NIH RePORTER database (1985–2019).
• Selection Criteria: Projects funded between 2009 and 2019.
  • Intramural: Activity codes starting with 'Z'.
  • Extramural: All other activity codes.
• Final Sample: 1,594 intramural grants and 97,054 extramural grants (Total: 98,648 projects).
• Output Data: 621,138 associated papers (via PubMed/PMID). Papers with mixed funding were excluded to ensure clear attribution.
• Data Cleaning: Removed renewal projects, normalized activity codes, and used a Random Forest model to exclude non-research-oriented projects (e.g., administrative or clinical practice-only projects).

Metrics and Variables:
The study utilized five specific output metrics:

1. Number of Papers: Raw publication count.
2. Relative Citation Ratio (RCR): Field- and time-normalized citation impact.
3. Approximate Potential to Translate (APT): A machine-learning prediction of whether a paper will be cited by clinical articles.
4. Total Clinical Citation Counts: Actual citations from clinical literature.
5. Binary Clinical Citation: Whether a paper received at least one clinical citation.

Analytical Approach:

• Topic Modeling: Used Word2Vec and spectral clustering on titles/abstracts to categorize research topics (e.g., viral infection, cancer, human-focused).
• Cost-Effectiveness Analysis: Calculated the cost per unit of output (e.g., cost per paper, cost per RCR). Costs were adjusted for inflation (2015 price level) and included indirect costs for extramural grants.
• Regression Analysis: Linear regression models controlled for Principal Investigator (PI) history (past publications, collaborations), project duration, and topic fixed effects to isolate the effect of funding type.
• Propensity Score Matching (PSM): Conducted 1:1 and 1:4 matching to pair intramural grants with the most similar extramural grants based on topic, PI history, and collaboration, reducing selection bias.
• Sensitivity Analysis: Inflated extramural indirect costs by 30% to account for unrecovered university expenditures.

3. Key Contributions

• Comprehensive Comparative Framework: This is one of the first large-scale studies to directly compare the efficiency and output profiles of intramural vs. extramural funding using a unified dataset of nearly 100,000 grants.
• Differentiation of Mission Alignment: The study provides evidence that funding mechanisms are not interchangeable; they possess distinct comparative advantages aligned with their institutional incentives.
• Methodological Rigor: The use of PSM and robust regression controls for confounding variables (like PI experience and research topic), ensuring that observed differences are attributable to the funding mechanism rather than researcher selection.
• Clinical Translation Focus: By introducing metrics specifically tied to clinical impact (APT and clinical citations), the study moves beyond traditional bibliometrics to assess "bench-to-bedside" value.

4. Key Results

Funding and Topic Differences:

• Grant Size: Intramural grants are significantly larger on average ($0.42M–$0.45M/year) compared to extramural grants (<$0.5M/year), reflecting the lack of cost-sharing in intramural labs.
• Research Topics: Intramural research is overrepresented in viral infections, cancer, and genetics (driven by specific NIH centers like the Vaccine Research Center and NCI). Extramural research has a broader distribution, including more adolescent and maternal health studies.

Output Efficiency (Cost-Effectiveness):

• Academic Metrics: Extramural research is more cost-effective for producing traditional academic outputs. When adjusting for cost, extramural grants generate more papers and higher Relative Citation Ratios (RCR) per dollar spent.
• Clinical Metrics: Intramural research is more cost-effective for generating clinical impact. Intramural grants show a comparative advantage in "Approximate Potential to Translate" and actual clinical citations per dollar.
• Time Dynamics: The gap in academic productivity (papers/RCR) between the two groups narrowed over time, but the clinical advantage for intramural research remained consistent.

Robustness Checks:

• Propensity Matching: Results held true even when comparing intramural grants to the most similar extramural grants (1:1 and 1:4 matching), confirming that the differences are structural rather than due to PI selection bias.
• Indirect Costs: Even when inflating extramural indirect costs by 30% (to account for university subsidies), extramural research remained more cost-effective for academic metrics, while intramural research retained its advantage in clinical translation.
• Human vs. Animal Research: Contrary to the hypothesis that intramural research is more "human-focused," long-duration intramural projects actually showed lower human-research scores and higher animal/molecular scores than extramural projects. This suggests the clinical advantage of intramural research is not simply due to doing more human studies, but rather how that research is structured or translated.

5. Significance and Implications

• Portfolio Management: The findings suggest that NIH (and similar agencies) should not view intramural and extramural programs as redundant. Instead, they should be leveraged for their specific strengths:
  • Extramural: Best for maximizing the volume of knowledge generation, training the workforce, and producing high-impact academic literature.
  • Intramural: Best for high-risk, long-term projects specifically aimed at clinical translation and solving specific agency health missions.
• Policy Reform: The study informs recent Senate proposals regarding NIH reform. It suggests that shifting the intramural program toward basic research might dilute its unique strength in clinical translation. Conversely, encouraging extramural researchers to focus on basic science could optimize the overall portfolio.
• Institutional Incentives: The results highlight that institutional incentives drive performance. Universities (extramural) are incentivized to publish and cite (academic prestige), while NIH intramural labs are incentivized to align with agency health goals (clinical impact).
• Broader Applicability: These insights may apply to other federal agencies (e.g., Department of Defense, NSF) that utilize both internal labs and external grants, suggesting that aligning funding mechanisms with specific strategic goals (basic vs. applied) yields better returns.

In conclusion, the paper argues that a "one-size-fits-all" approach to research funding is suboptimal. A strategic portfolio that recognizes the distinct comparative advantages of intramural (clinical translation) and extramural (academic volume/impact) mechanisms will maximize the return on public investment in science.