Effects of Early Career Peer Review Service on Subsequent Grant Submission Outcomes&nbsp

This study utilizes propensity score matching on NIH administrative data to demonstrate that early career peer review service is associated with increased grant submission volume, improved review scores, and higher funding success rates compared to non-reviewing peers.

The Gist

The Big Question: Does "Being a Judge" Make You a Better "Contestant"?

Imagine a massive, high-stakes cooking competition. To win a prize (a research grant), you have to submit a recipe (a grant application). But before your recipe is judged, you have to wait in line while a panel of expert judges tastes and critiques hundreds of other recipes.

For years, many chefs (scientists) have claimed: "If I get to sit on the judging panel, I'll learn exactly what the judges are looking for. I'll see how they argue, what scores they give, and what makes a recipe 'discussable.' This experience will make my own recipes much better, and I'll win more prizes."

However, until this study, there was no hard proof. The problem was that the people invited to be judges were usually already the best chefs in the kitchen. They had won awards, published cookbooks, and had great reputations. So, if they won the competition later, was it because they were judges, or just because they were already talented?

The Experiment: A "Taste-Test" Match-Up

To solve this, the researchers (from the NIH, the giant organization funding these "cooking competitions") set up a clever experiment using a method called Propensity Score Matching.

Think of this like a dating app for scientists. They took two groups of people:

1. The Judges: Early-career scientists who actually served as reviewers (ECRs).
2. The Non-Judges: Early-career scientists who were just as qualified to be judges (they had the same number of publications, the same job titles, and the same experience) but never got the chance to serve.

The researchers used a computer algorithm to pair them up perfectly, like matching twins. One twin from the "Judge" group was paired with a twin from the "Non-Judge" group. This ensured that any differences in their future success couldn't be blamed on them being naturally more talented or having better resources.

What Happened Next? (The Results)

The researchers watched these two groups for three years after the "Judge" group finished their service. They looked at how many new recipes they submitted, how many got a second look (discussed), how high their scores were, and how many won prizes (funding).

Here is what they found:

• More Entries: The former judges submitted more recipes than their matched twins.
• Better Scores: Their recipes were more likely to get a "high score" (meaning the judges thought they were excellent).
• More Discussion: Their recipes were more likely to be picked for a lively group discussion, which is a big deal in these competitions.
• More Wins: Most importantly, the former judges won more funding prizes than the non-judges.

The "Secret Sauce" (Why did this happen?)

The paper suggests a few reasons why being a judge helped:

1. Insider Knowledge: Just like a contestant who has watched the judges argue for years, these scientists learned the "secret language" of the review process. They learned how to write their applications so they hit the right notes and avoided the common pitfalls.
2. Confidence: Seeing how the process works from the inside might have made them more confident in submitting more applications.
3. Networking: Sitting in the room with other experts might have opened doors for collaborations and mentorship, though the study notes this is hard to measure directly.

The Catch (Limitations)

The authors are careful to say that this doesn't prove that every judge becomes a winner.

• The "Average" Score: While the judges won more often, the average score of all their recipes only improved slightly. It's possible that only a few judges got a huge boost, while the rest stayed the same, but those few big wins pulled the group up.
• Hidden Factors: There might be other things we didn't measure (like how well they networked at a party) that helped them win, not just the review service itself.
• Not a Guarantee: Being a judge doesn't guarantee you get money. The final decision involves many other factors, like the organization's budget and priorities.

The Bottom Line

This study provides the first solid evidence that serving as a peer reviewer actually helps early-career scientists get more grants later.

It's like saying: "If you get to sit in the judge's chair, you learn the rules of the game so well that when you play as a contestant, you are more likely to win." The researchers conclude that this program is a great way to train the next generation of scientists, not just to help the organization run its meetings, but to help those scientists succeed in their own careers.

Technical Summary

Technical Summary: Effects of Early Career Peer Review Service on Subsequent Grant Submission Outcomes

Problem Statement
The National Institutes of Health (NIH) relies heavily on peer review to evaluate the scientific merit of grant applications, a process requiring significant uncompensated time from scientists. While many reviewers subjectively believe that serving on study sections improves their own grant writing skills and competitiveness, empirical evidence supporting this claim has been limited. A primary challenge in evaluating this relationship is selection bias: investigators selected for peer review typically possess characteristics (e.g., strong publication records, prior funding) that also predict successful grant outcomes. Consequently, it is difficult to distinguish whether better outcomes are caused by the review experience or simply reflect the pre-existing quality of the selected investigators. This study addresses this confounding by examining whether early career investigators who served as NIH Early Career Reviewers (ECRs) achieved better subsequent grant outcomes compared to a matched group of eligible investigators who did not serve.

Methodology
The study employed a retrospective, quasi-experimental design using administrative data from the NIH Center for Scientific Review (CSR) and publication data from Dimensions.

• Study Population: The treatment group consisted of 1,862 investigators who served as ECRs during the 2020–2021 council years. The control group was constructed from investigators who met ECR eligibility criteria (e.g., Assistant Professor rank, prior application review, specific publication counts, no prior R01 funding) but had not served as reviewers.
• Propensity Score Matching (PSM): To address non-random selection and confounding, the authors used 1:1 nearest-neighbor propensity score matching without replacement. Covariates included research field, sex, race/ethnicity, prior application volume, prior funding history, publication counts, academic age, and institutional funding levels. This process resulted in a balanced analytic sample of 1,120 ECRs and 1,120 matched controls.
• Outcomes: Outcomes were assessed over a three-year follow-up period following the index date (service date for ECRs; equivalent review date for controls). Primary outcomes focused on R01 and R01-equivalent applications and included: submission volume, number of discussed applications, number of applications receiving high impact scores (≤35), number of funded applications, total award amounts, and peer review scores.
• Statistical Analysis: Analyses utilized regression models adjusted for matching covariates. Sensitivity analyses included models further adjusting for submission volume to test if outcome differences were merely a function of increased application frequency.

Key Results

• Submission and Discussion: ECRs submitted significantly more R01 applications than controls (mean 2.52 vs. 1.40) and were more likely to have applications discussed (mean 1.41 vs. 0.68).
• Scoring and Funding: ECRs achieved a higher number of applications with favorable impact scores (≤35) (0.47 vs. 0.20) and received more R01 awards (0.43 vs. 0.16). The effect sizes for these outcomes were moderate (Cohen's d ranging from 0.38 to 0.49).
• Regression Adjustments: Even after adjusting for the increased submission volume (Model 2), ECRs remained significantly more likely to receive at least one favorable score (Odds Ratio [OR] = 2.05) and at least one R01 award (OR = 2.58) compared to controls.
• Peer Review Scores: ECRs received slightly more favorable mean overall impact scores than controls, particularly for their last submitted application during the follow-up period, though the effect sizes for mean score differences were small to medium.

Key Contributions
This study provides the first large-scale, empirical evidence linking early career peer review service to improved grant application outcomes. By utilizing propensity score matching to rigorously control for pre-existing productivity and career stage, the authors isolate the association between review service and subsequent success. The findings validate the hypothesis that the "learning by doing" aspect of peer review—gaining exposure to scoring norms, panel discussions, and review criteria—translates into tangible improvements in grant competitiveness.

Significance and Claims
The paper claims that early career participation in NIH peer review is associated with improved grant application outcomes, including higher submission rates, better review scores, and increased funding success. The authors interpret these findings as consistent with the self-reported belief among scientists that review service enhances their ability to write competitive grants.

However, the authors maintain a modest tone regarding causal inference. They explicitly acknowledge that while the study design controls for measured confounders, unmeasured factors (such as networking opportunities or unquantifiable aspects of the selection process) could still influence the results. Furthermore, they note that peer review scores are not the sole determinant of funding, as second-tier review considers portfolio balance and scientific priorities. The study concludes that while causal claims are limited by the observational nature of the data, the consistency of positive effects across multiple outcomes and analytic approaches supports the value of the ECR program as a mechanism for scientific workforce development.