Challenges in the Computational Reproducibility of Linear Regression Analyses: An Empirical Study

This empirical study of 95 PLOS ONE health research papers reveals that over half of the analyzed linear regression analyses were not computationally reproducible due to poor variable mapping and inadequate methodological reporting, prompting a call for enhanced transparency through data dictionaries, analysis code, and a proposed Model Location and Specification Table (MLast).

The Gist

Imagine you and a friend are trying to bake the exact same cake. Your friend sends you a photo of their finished cake and says, "Here is the recipe I used!" But when you try to bake it yourself, you run into problems: the recipe says "a pinch of salt," but doesn't say which salt; it mentions "two cups of flour" but doesn't tell you if the cup was level or heaped; and the most confusing part, it lists ingredients like "flour A" and "flour B" without explaining which one is which.

You end up with a cake that looks nothing like the photo. You can't tell if your friend made a mistake, if the recipe was just vague, or if you just couldn't figure out the instructions.

This is exactly what this paper is about, but instead of cakes, it's about scientific research.

The Big Problem: The "Recipe" is Missing

The researchers looked at 95 health studies published in a journal called PLOS ONE. These studies used a common statistical tool called Linear Regression (think of this as a fancy way of drawing a line through a bunch of data points to see if two things are connected, like "does eating more vegetables lower blood pressure?").

The scientists wanted to see if other researchers could take the original data and the "recipe" (the methods) and get the exact same results. This is called Computational Reproducibility.

What They Found: A Kitchen Disaster

Here is the breakdown of their findings, using our baking analogy:

1. The "We Have Ingredients" Lie: Out of 95 papers, 68 authors claimed, "Yes, we have our ingredients (data) available for you to see!"
2. The Missing Ingredients: However, for 25 of those papers, the "ingredients" were either missing, broken, or labeled so poorly that no one could use them. It was like being handed a bag of flour that was actually sand.
3. The Failed Bake: The researchers picked 20 papers where the data was actually available and tried to recreate the analysis.
   • The Result: Only 8 out of 20 (40%) could be successfully reproduced.
   • The Failure Rate: That means 60% of the time, the researchers could not recreate the results, even though they had the data.

Why Did It Fail?

The main reason wasn't that the math was too hard; it was that the instructions were a mess.

• The "Which Flour?" Problem: The papers described variables (like "age" or "income") but didn't match them to the actual columns in the data file. It's like the recipe says "use the red bowl," but the data file has a bowl labeled "Container 4."
• The Hidden Steps: Authors often forgot to mention small but crucial steps, like "we threw out any data where the person was over 90" or "we adjusted the numbers for people who didn't answer this one question." Without these notes, the recipe is impossible to follow.

The Proposed Solution: A Better Recipe Card

The authors aren't just complaining; they are offering a solution to fix this kitchen disaster.

1. The Data Dictionary (The Ingredient List): When scientists share their data, they must provide a "Data Dictionary." This is a simple list that says, "Column A in the file is 'Age', Column B is 'Blood Pressure'." No guessing allowed.
2. The Code and Steps: They need to share the actual computer code they used, not just the final numbers.
3. The "MLast" Table (The New Recipe Card): This is the paper's big idea. They propose a new table called the Model Location and Specification Table (MLast).
   • Think of this as a GPS for the analysis.
   • It would clearly map out: "We used Data from Row 10 to Row 500, we removed these specific people, and we ran this specific test."
   • It connects the story in the paper directly to the numbers in the file, so anyone can follow the path from start to finish.

The Bottom Line

The study concludes that more than half of the health research they looked at is currently unreproducible. This is scary because if we can't recreate the results, we can't trust the conclusions. If a doctor makes a decision based on a study that can't be reproduced, it could affect real people's health.

To fix this, scientists need to stop treating their data like a secret recipe and start treating it like a public cookbook: clear, labeled, and easy for anyone to follow.

Technical Summary

Technical Summary: Challenges in the Computational Reproducibility of Linear Regression Analyses

1. Problem Statement
Despite growing initiatives to share research data, a significant gap remains between data availability and the actual computational reproducibility of published findings in health research. While many papers claim data availability, the usability of these datasets for independent replication is often compromised. Specific barriers include:

• Inadequate Reporting: Methods sections frequently lack sufficient detail regarding variable adjustments, data exclusions, and specific model specifications.
• Variable Mismatch: There is a critical disconnect between the variable names/descriptions in the manuscript and the actual column headers or labels in the shared raw data.
• Missing Artifacts: The absence of analysis code, data dictionaries, and model specifications prevents researchers from re-running the exact statistical tests (specifically linear regression) to verify results.

2. Methodology
The study employed an empirical assessment of computational reproducibility using a random sample of 95 health research papers published in PLOS ONE in 2019 that utilized linear regression.

• Data Accessibility Screening: The authors first assessed data availability. While 68 papers (71.6%) reported data availability, only 43 papers actually provided accessible data.
• Reproducibility Audit: From the accessible subset, the first 20 papers were selected for a deep-dive reproducibility audit.
• Re-analysis Protocol: For each of the 20 papers, the researchers attempted to re-analyze three distinct regression models per paper using the shared data and the methods described in the text. Success was defined as achieving identical results to the published findings.

3. Key Results
The study revealed a stark failure rate in computational reproducibility:

• Data Availability vs. Usability: Of the 68 papers claiming data availability, 25 (37%) lacked the necessary data to reproduce the specific linear regression models, rendering the claim of availability practically useless for replication.
• Reproducibility Rate: Only 8 out of 20 (40%) of the audited papers were successfully reproduced. Consequently, 60% of the analyzed studies were deemed not computationally reproducible.
• Primary Failure Points:
  • Variable Mapping: The most significant barrier was the inability to map variables described in the text to the actual data columns due to missing or inconsistent labeling.
  • Methodological Opacity: Critical details regarding data exclusions, specific variable adjustments, and model specifications were often omitted from the text.

4. Key Contributions and Proposed Solutions
To address these systemic issues, the paper proposes a structured framework for improving transparency:

• Mandatory Data Dictionaries: Authors must provide a data dictionary where variable labels in the dataset explicitly match the terminology used in the manuscript.
• Comprehensive Artifact Deposition: Analysis code, full model specifications, and supporting materials detailing every step of the analysis should be deposited in public repositories or included as supplementary files.
• The "MLast" Framework: The authors introduce a novel tool called the Model Location and Specification Table (MLast).
  • Function: MLast serves as a structured table that tracks exactly where (which section/file) and what (specific parameters, variables, and adjustments) analyses were performed.
  • Impact: When paired with a data dictionary, MLast enables the precise mapping of analytical steps to data structures, effectively bridging the gap between the manuscript and the raw data to facilitate automated or manual reproduction.

5. Significance
This study provides empirical evidence that current data-sharing practices in health research are insufficient to ensure the credibility of reported statistical results. The finding that 60% of studies cannot be reproduced undermines the foundation of evidence-based policy and practice. By highlighting the specific technical failures (variable mismatch, lack of code) and offering a concrete solution (MLast + Data Dictionaries), the paper advocates for a shift from mere "data availability" to "data usability." Implementing these standards is crucial for advancing open science, restoring trust in health research, and ensuring that shared data genuinely supports independent verification and collaboration.