Report on reproducibility in condensed matter physics

This report presents collaborative recommendations from a diverse group of experts to improve reproducibility and replicability in condensed matter physics by outlining best practices and policies across all stages of the scientific process.

The Gist

Imagine Condensed Matter Physics (CMP) as a massive, bustling kitchen where scientists are the chefs. These chefs aren't just making soup; they are inventing the ingredients for the future—like the chips in your phone, the screens on your TV, and the MRI machines in hospitals. For decades, this kitchen has been a place of incredible innovation.

However, a group of experts recently gathered to discuss a growing problem: The "Recipe" Crisis.

In science, a "reproducible" result means that if Chef A writes down a recipe for a perfect soufflé, Chef B should be able to buy the same ingredients, follow the same steps, and make the exact same soufflé. If Chef B can't do it, the recipe is broken.

This report argues that in the world of physics, too many recipes are missing steps, written in secret code, or so vague that no one else can follow them. Here is a simple breakdown of what the paper says, using some kitchen metaphors.

1. The Problem: "Magic" Soufflés

The report says that while physics is often thought of as a "hard" science where facts are solid, it is actually suffering from a Replication Crisis.

• The "Hype" Menu: Chefs (scientists) are under pressure to serve the most exciting, flashy dishes to get famous and get funding. A "normal" cake that works perfectly is boring. A "room-temperature superconductor" (a cake that floats and never melts) is a headline.
• The Missing Ingredients: To get that headline, some chefs might leave out the boring details in their recipe. They might say, "Add a pinch of magic," without explaining what the magic is. They might show a picture of the perfect soufflé but hide the fact that they tried 50 times and only kept the one that worked.
• The "Secret Sauce": Sometimes, chefs keep their special techniques or computer code secret because they think it gives them a competitive edge. But if you don't share the sauce, no one can taste the dish to see if it's actually good or just a trick.

2. The Difference Between "Reproducible" and "Replicable"

The paper makes a helpful distinction between two terms:

• Reproducibility (The "Copy-Paste" Test): You have the original data and the code. You run the same computer program on the same computer. If you get the same numbers, it's reproducible. It's like using a digital recipe app to bake the exact same cake.
• Replicability (The "New Kitchen" Test): You go to a different lab, buy your own ingredients, and build your own oven from scratch. If you can still make the cake, it's replicable. This is the ultimate test. The report says we are failing at this because we don't share enough of the "ingredients" (raw data, materials, and full instructions).

3. Why is this happening? (The Incentive Trap)

The report explains that the system is broken because of rewards.

• The Trophy vs. The Truth: If you publish a groundbreaking, flashy discovery, you get a trophy (a job, a grant, fame). If you publish a boring paper saying, "I tried to copy that other guy's discovery and it didn't work," you get nothing. In fact, you might get fired.
• The "Scooping" Fear: Chefs are afraid that if they share their secret recipe too early, someone else will steal it and get the credit first. So, they hoard their data.
• The "Publish or Perish" Pressure: Universities and funding agencies want big results. They don't want to pay for the boring work of checking if the big results are real.

4. The Solution: A New Menu for Science

The group of experts (scientists, journalists, lawyers, and publishers) came up with a list of rules to fix the kitchen. Here are the main ideas:

For the Chefs (Scientists):

• Share the Whole Pan: Don't just show the pretty picture of the soufflé. Share the raw data, the messy notes, and the code you used.
• Tell the Whole Story: If you tried 100 samples and only 1 worked, say so! Don't hide the 99 failures.
• The "Full Team" Check: Before you publish, every person on the team should be able to look at the data and say, "Yes, I understand how we got this result."

For the Restaurant Owners (Publishers/Journals):

• Stop Judging by the Cover: Don't just accept papers because they sound exciting. Check if the recipe actually works.
• Publish the "Failures": Create a special section in the menu for papers that say, "We tried to copy this, and it failed." This is valuable information!
• The "Open Kitchen" Policy: Require that all data and code be available in a public digital pantry before the paper is published.

For the Investors (Universities & Funding Agencies):

• Reward the Cleaners: If a scientist spends time checking someone else's work or fixing a mistake, give them credit. Treat "cleaning the kitchen" (reproducibility) as important as "cooking the meal."
• Fund the Checkers: Give money specifically to people whose job is to try and copy other people's results to see if they are true.

The Bottom Line

The report is a wake-up call. It says that science is supposed to be a self-correcting machine, but right now, it's jammed. If we don't fix the way we share our "recipes," we risk building the future on shaky foundations.

By making science more open, transparent, and honest, we won't just stop fraud; we will actually speed up discovery. Imagine if every chef shared their best tricks with everyone else—the whole world would eat better, faster. That is the goal of this report: to turn physics from a competition of secrets into a collaboration of truth.

Technical Summary

Based on the provided report, here is a detailed technical summary of the paper "Report on reproducibility in condensed matter physics."

1. Problem Statement

Condensed Matter Physics (CMP), a field responsible for foundational technologies (e.g., semiconductors, MRI, LEDs), is facing a growing reproducibility and replicability crisis. While physics is often perceived as immune to such issues due to the invariance of natural laws, the report argues that CMP suffers from systemic challenges similar to those in social and biomedical sciences.

Key Drivers of the Problem:

• Incentive Structures: A "publish or perish" culture prioritizes novel, high-impact, and sensational claims over mundane, negative, or confirmatory results (termed the "Inverse Occam's Razor"). This creates pressure to present data in ways that favor exciting narratives.
• Lack of Transparency: Published works often lack sufficient detail regarding experimental parameters, raw data, code, and sample preparation to allow independent verification.
• Ambiguity in Definitions: The distinction between reproducibility (recreating results using the same data/code) and replicability (obtaining consistent results via new experiments/data) is often blurred, hindering clear policy formulation.
• Barriers to Verification: Concerns include data hoarding, proprietary code, fear of being "scooped," and institutional reluctance to investigate misconduct due to reputational risks.
• Specific High-Profile Failures: The report cites recent retractions and failed replications in areas such as room-temperature superconductivity, Majorana zero modes, and high-pressure superconductivity as evidence of the crisis.

2. Methodology

The report is not an experimental study but a consensus document derived from a collaborative working group.

• Event Basis: The conclusions stem from the International Conference on Reproducibility in Condensed Matter Physics, held at the University of Pittsburgh in May 2024.
• Stakeholder Composition: The working group included over 30 experts from diverse sectors:
  • Academic researchers (experimentalists and theorists).
  • Government laboratory scientists.
  • Scientific journalists and legal professionals.
  • Representatives from publishers (e.g., APS, Nature, Science) and professional societies.
• Process: The group engaged in 7 hours of structured discussions over three days, analyzing case studies (e.g., Majorana fermions, density functional theory) and identifying systemic failures. The report synthesizes these discussions into a set of actionable recommendations.

3. Key Contributions and Recommendations

The report provides a comprehensive framework for improving scientific integrity in CMP, categorized by stakeholder group.

A. Best Practices for Researchers

• Data & Code Sharing: Release all primary data, analysis scripts, and code in citable repositories. Strive for FAIR (Findable, Accessible, Interoperable, Reusable) standards, but share data in its current form if full FAIR compliance is not immediately possible.
• Full Parameter Disclosure: Report the full range of experimental parameters explored, not just the subset that yields "successful" results, to mitigate confirmation bias.
• Authorship Accountability: All authors must have access to and review primary data and analysis files before publication.
• Comprehensive Reporting: Manuscripts must include all reasonable data presentations (e.g., different scales, raw vs. processed) and discuss all plausible alternative interpretations of results.
• Sample Reporting: Explicitly state the number of samples/devices tested and report on non-confirmatory data.
• Correction Culture: Authors must promptly retract or correct work if errors are discovered post-publication without stigma.

B. Recommendations for Publishers

• Shift Evaluation Criteria: Prioritize scientific validity, accuracy, and data quality over subjective "novelty" or "flashiness."
• Mandatory Data Availability: Replace "data available upon request" with mandates for complete data and scripts in public repositories.
• Replication Support: Create specific article types or sections for replication studies and negative results. Establish clear criteria for accepting such papers.
• Review Process: Implement open review processes (open reports, open identities) and require specific review of Supplemental Material.
• Retraction Transparency: Draft clear, transparent policies for retractions and corrections to reduce the stigma associated with them.

C. Recommendations for Universities and Labs

• Redefine Outputs: Treat datasets and code as primary research outputs equivalent to publications in tenure and promotion reviews.
• De-emphasize Metrics: Move away from journal impact factors and citation counts as primary metrics for evaluating individual scientists.
• Misconduct Investigation: Establish independent, transparent, and rigorous procedures for investigating allegations of misconduct and unreliable science (beyond just fabrication/falsification/plagiarism).
• Whistleblower Protection: Actively protect junior researchers and whistleblowers who raise concerns about data integrity.

D. Recommendations for Funding Agencies

• Fund Replication: Explicitly fund projects dedicated to replicating high-profile claims, particularly those that form the basis for further funding decisions.
• Grant Requirements: Mandate that grant proposals include detailed plans for data management, sharing, and reproducibility.
• Reward Negative Results: Create mechanisms to reward research that yields negative results or undermines previous claims.
• Oversight: Require institutions to report allegations of misconduct to funding agencies and ensure external experts are involved in investigations.

4. Results and Findings

The report concludes that the current state of CMP is not immediately reproducible or replicable for the majority of published works due to insufficient data sharing and reporting standards.

• Prevalence: While exact statistics are hard to pin down due to underreporting, surveys indicate significant concern within the community (e.g., 12% of junior APS members witnessed "less than truthful" descriptions of techniques).
• Systemic Nature: The crisis is not merely due to individual malfeasance but is driven by structural incentives that favor speed and novelty over rigor.
• Feasibility: The technical infrastructure for open science (repositories like Zenodo, Figshare) exists and is adequate for most CMP data; the barrier is cultural and policy-driven.

5. Significance

This report is a landmark intervention for the condensed matter physics community.

• Cultural Shift: It moves the conversation from "physics is immune to the replication crisis" to acknowledging specific vulnerabilities in CMP, such as bespoke equipment and small sample sizes.
• Policy Blueprint: It provides a concrete, multi-stakeholder roadmap for reforming how research is conducted, published, and funded.
• Long-term Impact: By advocating for FAIR data, open review, and the de-stigmatization of replication/negative results, the report aims to restore public trust, optimize the use of public funding, and ensure that future technological advances are built on a solid, verifiable foundation.
• Subfield Specificity: It calls for the development of subfield-specific reporting standards (e.g., for topological materials or superconductivity), acknowledging that a "one-size-fits-all" approach may not work for the diverse techniques in CMP.