Beyond Reproducible Research: Building a Formal Representation of a Data Analysis

This paper argues for and presents an implementation of a formal representation of data analysis that externalizes an analyst's logical reasoning and assumptions, thereby enabling evaluation, visualization, and sensitivity testing of conclusions without relying solely on executable code or raw data.

The Gist

Imagine you are a chef who has just created a delicious new soup. You write down the recipe (the code) and list the ingredients (the data). You hand this to a friend and say, "Here, try to make this soup exactly like I did." This is the current standard for Reproducible Research.

However, there's a problem. Your friend might follow the recipe perfectly, but they still don't know why you chose those specific ingredients.

• Did you add salt because you wanted it salty, or because you thought the tomatoes were too acidic?
• Did you ignore the burnt carrots because you thought they were just "roasted," or because you didn't notice them?
• What if the soup tastes weird because you used a specific type of water that you didn't mention?

The recipe (code) tells them what you did, but it hides why you did it and what you were expecting to happen.

The Problem: The "Black Box" of Data

In the world of data science, researchers often just share their code and the final numbers. If the code runs without crashing, everyone assumes the result is correct. But this is like judging a magic trick just because the rabbit appeared. You don't know if the magician actually pulled the rabbit out of a hat or if they had a second rabbit hidden in their sleeve.

If the data has a hidden flaw (like a missing number or a weird outlier), the code might still run, but the conclusion could be wrong. Because the code doesn't explicitly state the researcher's assumptions (e.g., "I assume there are no missing numbers"), these errors can slip through silently.

The Solution: The "Logical Blueprint"

Roger Peng, the author of this paper, proposes a new way to share data analysis. Instead of just sharing the "recipe," he suggests sharing a Logical Blueprint or a Proof.

Think of it like building a house.

• Current Method: You show someone the finished house and the list of tools you used. They can try to build it, but they don't know if the foundation is solid or if you skipped a step because you "thought" the ground was level.
• Peng's Method: You give them a blueprint that says: "To build this roof, we first proved the foundation is flat. To prove the foundation is flat, we checked that the ground has no holes. To prove there are no holes, we shined a light on the ground."

In this new system, every claim you make (e.g., "The average height is 5 feet") must be backed up by a chain of evidence. You have to explicitly state:

1. The Claim: "The average is 5 feet."
2. The Premise: "This is true ONLY IF there are no missing numbers."
3. The Evidence: "I checked the data, and there are indeed no missing numbers."

How It Works (The "Class" System)

The paper uses a programming concept called "Classes" (which you can think of as Identity Badges).

Imagine every statement you make in your analysis gets a badge.

• Badge A: "No Missing Numbers."
• Badge B: "No Extreme Outliers."
• Badge C: "The Average is 5 Feet."

In Peng's system, you cannot wear Badge C (The Average is 5 Feet) unless you are also wearing Badge A and Badge B. The code forces you to check for the missing numbers and outliers before you are allowed to claim the average.

If the data has missing numbers, the system simply refuses to give you the "Average is 5 Feet" badge. It stops you from making a false claim.

Why Is This Better?

1. You Don't Need the Data to Check the Logic
With the old method, to check if a study is right, you have to download the massive dataset and run the code again. It's like having to bake the whole cake just to see if the recipe makes sense.
With Peng's blueprint, you can look at the "Logical Tree" and see: "Ah, they claimed the average is 5 feet. They said this is only true if there are no outliers. But wait, their data does have outliers! Therefore, their claim is unsupported." You can spot the flaw without ever seeing the actual data.

2. It Stops "Silent Errors"
Sometimes, data is messy. Maybe "USA" is written as "US" in one file and "USA" in another. A computer might join them and accidentally delete half the data, but it won't scream an error; it will just give you a wrong answer.
In Peng's system, you would have a badge that says "The joined file must have 100 rows." If the join fails and you only get 50 rows, the system screams, "ERROR! You don't have the right badge!" It forces you to fix the data before you can proceed.

3. It Visualizes the Reasoning
The paper suggests drawing a tree diagram of these badges.

• Top Branch: "The Drug Cures the Disease."
• Middle Branch: "Because the patients improved."
• Bottom Branch: "Because the patients didn't have other illnesses" AND "Because the measurement tool was accurate."

If you look at the tree, you can instantly see if the logic holds up. If one of the bottom branches is weak, the whole tree falls over.

The Catch

The author admits this is a lot of work. Writing these "badges" and "proofs" takes more time and code than just writing a simple script. It's like writing a legal contract for every step of your cooking. It's verbose and tedious.

However, the paper argues that this extra effort is worth it. It forces scientists to think clearly about what they are doing, exposes their assumptions, and makes it much harder to hide mistakes. It turns data analysis from a "black box" into a transparent, logical argument that anyone can inspect, even without running the code.

Summary

• Old Way: "Here is my code and my result. Trust me, it works."
• New Way: "Here is my result. Here is the logical proof that the result is valid, including every assumption I made and every check I performed. If you check my logic, you will see why this result is true."

It's the difference between a magician saying "Abracadabra!" and a scientist showing you the hidden trapdoor, the spring mechanism, and the proof that the rabbit was never actually in the hat to begin with.

Technical Summary

Here is a detailed technical summary of the paper "Beyond Reproducible Research: Building a Formal Representation of a Data Analysis" by Roger D. Peng.

1. Problem Statement

The paper addresses the limitations of the current standard for reproducible research, which typically involves sharing raw data and the imperative code used to analyze it. While this allows others to re-run analyses, the author argues that it fails to capture the logical reasoning, premises, and assumptions of the data analyst.

Key issues identified include:

• Hidden Reasoning: Code shows what was done, but not why specific steps were taken or what the analyst expected the data to look like.
• Static vs. Dynamic: Reproducibility is a dynamic process (running code); it does not provide a static proof of the validity of the analysis statements.
• Verification Burden: Verifying results requires re-executing code on potentially sensitive or large datasets, which is resource-intensive.
• Silent Errors: Imperative code often executes without error even when the logic is flawed (e.g., joining data frames with mismatched keys), leading to incorrect conclusions that are hard to detect without deep domain knowledge.
• Lack of Formal Validation: There is no universally agreed-upon method to present "proof" that a specific data analysis statement (e.g., "the mean is 4.6") is true, distinct from the mathematical proof of a theorem.

2. Methodology

The author proposes a Formal Static Representation of data analysis, treating data analysis statements not as program outputs, but as logical assertions supported by evidence. The methodology is implemented using the S4 class system in R, leveraging object-oriented programming principles to enforce logical constraints.

Core Principles

1. Data Analysis as Statements with Evidence: Instead of viewing analysis as a function taking input data and producing output, the analysis is viewed as a series of statements that require evidence (premises) to be considered true.
2. Statements as Class Definitions: A specific data analysis statement (e.g., "The mean of column 1 is 4.6") is encoded as an S4 class. An object of this class can only be instantiated if the statement is true according to a defined validity method.
3. Premises as Class Extensions: Supporting evidence is encoded as "premise classes" that extend the main statement class. These premises act as slots within the parent class.
   • Direct Approach: Identifying statements that directly imply the conclusion.
   • Indirect Approach: Identifying conditions that would make the statement false (alternative hypotheses) and proving they do not exist (e.g., proving no outliers exist to support a mean calculation).
4. Logical Hierarchy: The relationship between a conclusion and its premises forms a tree structure where child nodes (premises) must be true (logical AND) for the parent node (conclusion) to be valid.

Implementation Mechanics

• S4 Classes: Used to define the structure of data analysis statements.
• Validity Methods: Functions attached to classes that check if the data meets the criteria. If the check fails, object creation throws an error.
• Coercion: Methods to convert raw data (e.g., CSV columns) into these specialized classes, ensuring data is validated at the point of ingestion.
• Static Analysis: Because the logic is embedded in the class definitions, one can inspect the code to understand the analyst's expectations without running the code on the actual data.

3. Key Contributions

The paper introduces a framework that shifts the paradigm from "reproducible code" to "verifiable logical construction."

• Formalization of Reasoning: It provides a mechanism to externalize the analyst's assumptions (e.g., "no missing values," "symmetric distribution") directly into the code structure.
• Static Evaluation: It enables the evaluation of an analysis's quality and logic without executing the code or accessing the data. One can verify if the code logically supports the conclusion by inspecting the class hierarchy.
• Visualization of Logic: The hierarchical structure of premises can be visualized as a tree (similar to a fault tree in systems engineering), making the logical flow of the argument transparent to reviewers.
• Sensitivity Analysis: The framework allows for testing the sensitivity of conclusions to unexpected data features by simulating data that violates specific premise classes.
• Error Prevention: By defining expected outputs (e.g., row counts, column names) as validity checks, the system can catch "silent errors" (like failed data joins) immediately upon object creation.

4. Results and Examples

The paper demonstrates the approach through several R-based examples:

• Missing Data Check: A class vector_no_missing is defined. If a CSV is read into this class, the mere existence of the object proves (statically) that the column contains no NA or -99 values, without needing to inspect the data values.
• Mean Calculation: To claim "Mean = 4.6," the author constructs a class that requires three premises:
  1. No missing values (preventing NA mean).
  2. Median is close to 4.6 (preventing distribution shift).
  3. No significant outliers (preventing skew).
     Only if all three premise objects exist can the final "Mean = 4.6" object be created.
• Data Joining: A merged_dataset class is defined with strict validity checks (e.g., "must have 3 rows," "no NA in value columns"). When attempting to join mismatched data (e.g., "US" vs. "USA"), the validity check fails, returning an error immediately, whereas standard code would silently produce a 6-row or 0-row result.
• Linear Regression: A complex example shows a regression slope claim supported by premises regarding non-linearity, outliers, and visual residual plots (where the analyst manually confirms the plot looks "okay" via an interactive prompt).

5. Significance and Future Work

• Paradigm Shift: The paper argues that reproducibility is necessary but insufficient. The field needs a "semantic layer" on top of code to communicate the intent and logic of the analysis.
• Analogy to Systems Engineering: The premise trees are compared to Fault Trees used in aerospace and nuclear engineering, but inverted. While fault trees map paths to failure, these trees map paths to success (validity).
• Limitations: The current implementation using S4 classes is verbose and creates significant coding overhead, making it impractical for routine use without tooling improvements.
• Future Directions:
  • Developing more efficient syntax or DSLs (Domain Specific Languages) to reduce boilerplate.
  • Formalizing the validation of graphical summaries (e.g., using image hashing or SVG comparison) to automate the "visual check" premise.
  • Integrating this logic into standard statistical software to make "logical construction" a default part of the analysis workflow.

In conclusion, Peng proposes that by treating data analysis statements as formal, verifiable objects with explicit logical dependencies, the statistical community can move beyond merely reproducing results to rigorously evaluating the reasoning behind them.