Towards Automatically Inferring Constraints to Identify Implicit Assumptions in Data Analysis

Technical Summary: Towards Automatically Inferring Constraints to Identify Implicit Assumptions in Data Analysis

Problem Statement
High-level data analysis languages (e.g., R, Python) and notebook environments (e.g., Jupyter, Quarto) are widely used by domain experts who often lack formal software engineering training. Consequently, analysis artifacts (scripts and notebooks) frequently suffer from severe reproducibility and reusability issues. Studies indicate that 65% to 77% of analyzed artifacts fail to execute due to missing dependencies, incorrect execution orders, or undocumented setup steps. Furthermore, code reuse is hindered by the fact that a significant portion of shared code snippets are clones containing no unique logic, often failing when applied to different datasets or contexts.

The authors frame these difficulties not merely as technical bugs, but as implicit assumptions made by the original authors. These assumptions range from specific package versions and interpreter constraints to the expected shape of input data and manual setup steps. Existing tools for reproducibility often address specific symptoms (e.g., environment reconstruction or linting) but fail to provide a unified, systematic approach to identify, document, and validate these implicit assumptions, particularly regarding their interactions.

Methodology
The paper proposes a unified perspective that treats implicit assumptions as code constraints which can be identified and made explicit through static analysis. The core methodology involves:

1. Constraint Inference Framework: The authors differentiate between two analysis scenarios:

   • Preemptive Analysis: Conducted within the original execution environment where data and dependencies are known. The goal is to ensure the script is self-contained and compliant with FAIR principles.
   • Retroactive Analysis: Conducted when the original environment is unknown or data is partially available. The goal is to infer constraints to reconstruct executable environments or validate reuse in new contexts (reproduction, replication, or reuse).

2. Static Analysis Techniques: The approach leverages abstract interpretation and dataflow analysis to infer values, types, and dependencies without executing the code. The authors utilize flowR, a static program analysis framework for R, as a proof-of-concept implementation.

3. Three-Step Inference Strategy:

   • Inferring Package Versions (A1–A3): The system identifies all dependencies (libraries loaded directly, via ::, or indirectly) and tracks function signatures, class usage, and global object initialization. By comparing these against available package APIs and dependency trees, the system infers minimum version constraints for packages and the interpreter.
   • Identifying Script Interdependencies (A4, A7): The authors propose building a program dependency graph for each script, following explicit source calls and tracking global variables. This allows the system to link undefined variables to definitions in other scripts, identify file read/write dependencies, and determine valid topological execution orders.
   • Inferring Data Shape Expectations (A5, A6, A8–A10): Using new abstract domains, the system tracks the shape of data frames (column names, types, counts) as they are manipulated. It infers expectations for statistical functions (e.g., a t-test requiring a factor with two levels) and inserts dynamic guards (checks) into the artifact to validate data conformity at runtime.

Key Contributions

• Unified Perspective: The paper introduces "implicit assumptions" as a comprehensive framework for categorizing reproducibility challenges in data science, moving beyond isolated fixes to a holistic view of constraints.
• Static Analysis Approach: It proposes using static analysis (specifically abstract interpretation) to automatically infer these assumptions and transform them into explicit code constraints.
• Proof-of-Concept Implementation: The authors provide an explorative implementation on top of flowR. They evaluated this approach on 8,160 real-world R projects (33,336 files) scraped from repositories like Zenodo and Harvard Dataverse.
• Empirical Findings: The evaluation revealed that 14.6% of the analyzed projects contain implicit interdependencies between scripts, highlighting the prevalence of the problem the methodology aims to solve.

Results and Limitations
The paper presents a "proof-of-concept" rather than a fully matured production tool. The initial implementation successfully demonstrates the feasibility of inferring library requests, tracking data frame shapes, and identifying script interdependencies. However, the authors explicitly acknowledge limitations:

• Rice's Theorem: The inference of implicit assumptions is inherently limited by Rice's theorem, meaning a universal solution is impossible; the approach relies on heuristics derived from analyzing reproducible and irreproducible artifacts.
• Scope: The current implementation focuses on R, though the authors argue the challenges are language-agnostic.
• Human-in-the-Loop: The system may detect assumptions it cannot automatically resolve, requiring human intervention.

Significance
The authors claim that their work offers a novel, unified perspective on data analysis reproducibility. By converting silent assumptions into explicit, machine-readable constraints, the approach aims to:

1. Automate Transformation: Convert non-executable scripts into executable artifacts by generating necessary environment specifications.
2. Dynamic Validation: Serve as runtime checks to ensure data and environment conditions are met before execution.
3. Enhanced Documentation: Act as living documentation to support code comprehension and reuse across different projects and datasets.

The paper concludes that while the detection of implicit assumptions is a significant step, future work must focus on refining heuristics, expanding coverage to all assumption types, and evaluating the utility of these constraints through user studies and fuzzing of datasets.