Rigor and Transparency in two neurotrauma-publishing journals: editorial policies improve transparent reporting.

This study demonstrates that the *Journal of Neurotrauma*'s mandatory rigor section significantly improved the reporting of specific transparency items compared to *Experimental Neurology*, though it highlighted a need for further improvements in reporting sex as a biological variable and reagent details.

The Gist

Imagine the world of medical research as a massive construction site. Scientists are the architects and builders trying to design a cure for Traumatic Brain Injury (TBI), a condition that affects millions of people. For decades, they've built many blueprints (lab studies) that looked perfect on paper. But when they tried to build the actual skyscrapers (human clinical trials), the buildings kept collapsing.

Why? Because the blueprints were missing crucial details. They didn't say who was watching the measurements, how many bricks were needed, or exactly what kind of cement was used. This is the problem of rigor and transparency.

This paper is like a report card given to two different construction firms (journals) to see which one is teaching its builders to write better blueprints.

The Two Construction Firms (The Journals)

The researchers compared two major journals that publish TBI studies:

1. Journal of Neurotrauma (JNeuroT): This journal recently decided to change the rules. Starting in 2022, they told authors: "You must fill out a special 'Rigor Checklist' before we print your paper. You have to explicitly state how you blinded your study, calculated your sample size, and shared your data."
2. Experimental Neurology (ExpNeuro): This journal is similar in size and reputation, but they did not change their rules. They still just ask for a standard paper without a mandatory checklist.

The Detective Tool (SciScore)

To grade these papers, the authors didn't read every single word manually (that would take forever!). Instead, they used an AI detective named SciScore.

Think of SciScore as a super-fast robot inspector that scans thousands of blueprints in seconds. It looks for specific "safety tags" (like "blinding," "power calculation," or "data availability"). If a blueprint has the tag, it gets a point. If it's missing, it gets zero. The robot then gives the paper a score from 1 to 10.

What Did They Find?

The results were a mix of "Great job!" and "Not quite there yet."

1. The Checklist Works (For What It Asks)
When the Journal of Neurotrauma authors were forced to fill out the special checklist, they actually did it!

• The Analogy: Imagine a teacher tells students, "You must write a paragraph about your homework." Suddenly, 50% of the class writes that paragraph. Before, only 10% did.
• The Result: Papers with the new checklist were much better at reporting things like blinding (hiding who got the real treatment), power calculations (making sure the study was big enough), and data sharing. They scored significantly higher than the other journal.

2. The "Blind Spot" (What the Checklist Missed)
Here is the catch: The checklist was very specific. It asked for A, B, and C. But it didn't explicitly ask for D, E, and F.

• The Analogy: Imagine the teacher says, "Write about your homework." The students write about math and science. But they forget to mention history, even though history is just as important.
• The Result: Because the checklist didn't explicitly ask for Sex as a Biological Variable (reporting if the test subjects were male or female) or Transparent Antibody Reporting (naming exactly which chemical tools were used), the authors didn't report them. In fact, the papers with the checklist were sometimes worse at reporting these missing items than the papers from the other journal that didn't have a checklist at all!

3. The "False Sense of Security"
The authors suggest that when scientists see a checklist, they might think, "Oh, I filled out the form, so my study is rigorous!" They might stop paying attention to the other important details that weren't on the list. It's like a driver who puts on a seatbelt but then drives 100 mph because they feel "safe."

The Big Takeaway

The study concludes that forcing authors to fill out a specific section works, but only for the specific things listed on that section.

• Good News: The Journal of Neurotrauma is leading the way by requiring these sections, and it is improving reporting on the items they asked for.
• Bad News: It's not a magic wand. If the checklist doesn't ask for "Sex" or "Reagent IDs," authors often skip them.

The Final Lesson:
To fix the broken bridge between lab experiments and real cures, journals can't just add a checklist and call it a day. They need to make sure the checklist covers everything that matters (like sex differences and exact reagent names), not just the easy-to-see items. Otherwise, we're just building a house with a perfect front door, but the foundation is still shaky.

Technical Summary

1. Problem Statement

Preclinical research in Traumatic Brain Injury (TBI) has generated a vast amount of peer-reviewed literature, yet the translation of these findings to clinical settings has been minimal. A primary contributor to this "bench-to-bedside" failure is the lack of standardization, reproducibility, and transparency in experimental reporting. Key factors such as investigator blinding, randomization, power calculations, and the authentication of biological resources (e.g., antibodies, cell lines) are frequently under-reported or missing.

While major organizations like the NIH and various journals have introduced guidelines (e.g., ARRIVE, NIH rigor criteria) to address these issues, there is limited empirical evidence quantifying whether specific editorial policies actually change author reporting behaviors. Specifically, it is unclear if mandating a dedicated "Rigor and Transparency" section in manuscripts leads to measurable improvements in reporting compared to journals without such mandates.

2. Methodology

The study employed a comparative, quantitative analysis of two leading neurotrauma journals:

• Journal of Neurotrauma (JNeuroT): Implemented a mandatory "Transparency, Rigor, and Reproducibility" (TRR) section for submissions starting January 1, 2023.
• Experimental Neurology (ExpNeuro): A journal of similar size, impact factor, and topic scope, but without explicit TRR section requirements.

Data Collection and Processing:

• Dataset: Full-text research articles published between 2014 and 2024 deposited in PubMed Central (PMC).
• Groups:
  1. ExpNeuro (Control).
  2. JNeuroT (Pre-TRR policy).
  3. JNeuroT+ (Post-TRR policy, containing the extra rigor section).
• Tool: The study utilized SciScore (an AI-based tool, RRID:SCR_016251) to automatically parse manuscript text. SciScore detects specific sentences or phrases related to rigor criteria and assigns a score (1–10) based on adherence.
• Metrics: The analysis focused on the presence/absence of specific rigor items (e.g., blinding, power analysis, data availability, reagent identification) and calculated the proportion of papers reporting each item.
• Statistical Analysis: Two-sample Z-tests for population proportions were used to compare reporting rates between JNeuroT+, JNeuroT, and ExpNeuro ($p < 0.05$).

3. Key Contributions

• Quantitative Evaluation of Policy Impact: This study provides one of the first direct, large-scale measurements of how a specific editorial policy (the mandatory TRR section) influences reporting practices in a specific field.
• AI-Driven Scalability: By leveraging SciScore, the authors analyzed thousands of manuscripts, overcoming the labor-intensive limitations of manual chart reviews typically used in such studies.
• Granular Comparison of Guidelines vs. Practice: The study mapped the specific items requested in the JNeuroT TRR documents against what SciScore actually detected, revealing gaps between policy requirements and actual implementation (e.g., the policy asks for "materials sources," but SciScore requires specific catalog numbers/RRIDs for a "pass").

4. Key Results

Overall Scores:

• JNeuroT+ (papers with the TRR section) achieved the highest average SciScore (5.61), followed by JNeuroT without the section (4.60), and ExpNeuro (3.52).
• All groups scored higher than the general PubMed Central 2020 average (4.13), likely due to the inclusion of human studies which generally report better than preclinical animal studies.

Specific Rigor Items:

• Items Explicitly Requested in TRR: For items explicitly listed in the JNeuroT TRR documents, reporting rates in JNeuroT+ were significantly higher than in both control groups.
  • Power Analysis: Increased from ~8-11% (controls) to 47.06% in JNeuroT+.
  • Data Availability: Increased from ~6% to 51.35%.
  • Blinding: Increased from ~38-47% to 69.70%.
  • Attrition/Inclusion Criteria: Significant increases observed.
• Items NOT Explicitly Requested: For items not explicitly detailed in the TRR template, there was no significant improvement in JNeuroT+ compared to controls.
  • Sex as a Biological Variable: Reporting was actually significantly lower in JNeuroT+ (62.16%) compared to ExpNeuro (79.07%).
  • Antibody Transparency: Remained low (~11%) in both JNeuroT+ and ExpNeuro, far below the PMC average (31%).
  • Reagent Identifiability (RRIDs): Neither journal effectively mandated or achieved high rates of RRID usage for antibodies or cell lines.

Temporal Trends:

• The data shows a rapid adoption of the TRR section in 2023 and 2024. However, the 2024 data suggests a potential plateau or slight decline in scores for JNeuroT+, raising questions about the long-term sustainability of the improvement.

5. Significance and Implications

• Policy Effectiveness: The study confirms that mandatory, structured editorial sections are highly effective at increasing reporting for the specific items they request. Authors tend to "check the box" for required items but do not generalize this rigor to other important areas unless explicitly prompted.
• The "Checklist Paradox": The findings suggest a potential trade-off where focusing heavily on a specific checklist (the TRR section) may inadvertently lead authors to neglect other critical rigor factors (like sex differences or reagent authentication) that are not on the checklist.
• Gap in Reagent Transparency: Despite being a major NIH requirement, the identification of biological resources (via RRIDs or catalog numbers) remains poor in both journals. The study highlights that current editorial policies in these neurotrauma journals are insufficient for ensuring full FAIR (Findable, Accessible, Interoperable, Reusable) data principles regarding reagents.
• Future Directions: The authors argue that for true translational success, journals must move beyond "checklist compliance" to enforcing broader transparency standards, including mandatory reporting of sex as a biological variable and strict reagent identification, rather than relying solely on authors to self-report when not explicitly asked.

Conclusion:
The implementation of the TRR section in the Journal of Neurotrauma successfully improved reporting for targeted items (blinding, power, data availability). However, the policy did not improve, and in some cases worsened, reporting on untargeted but critical variables (sex, reagent ID). This indicates that while editorial mandates are a powerful tool, they must be comprehensive and continuously updated to prevent "tunnel vision" in scientific reporting.