💻 computer science

JunoBench: A Benchmark Dataset of Crashes in Python Machine Learning Jupyter Notebooks

This paper introduces JunoBench, the first benchmark dataset comprising 111 curated, reproducible real-world crashes and their fixes from public Kaggle notebooks, designed to advance research on debugging and repairing machine learning code in Jupyter environments.

Original authors: Yiran Wang, José Antonio Hernández López, Ulf Nilsson, Dániel Varró

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiran Wang, José Antonio Hernández López, Ulf Nilsson, Dániel Varró

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to bake a complex cake using a recipe book where you can write new notes, skip steps, or change the order of instructions as you go. This is what data scientists do when they use Jupyter Notebooks to build Machine Learning (AI) programs. It's flexible and fun, but because you can mix up the order of steps, things often go wrong. The cake might burn, the oven might explode, or the batter might turn into glue. These are called "crashes."

The problem is that when these crashes happen, it's very hard for computer programs (or even humans) to figure out why they happened and how to fix them. Why? Because there wasn't a good "practice test" available to teach debugging tools how to spot these specific mistakes.

Enter JunoBench. Think of JunoBench as a giant, organized training gym for crash-fixing robots.

Here is what the paper says about this new tool, broken down simply:

1. The Collection (The "Crash Library")

The researchers went out and found 111 real-life examples of these "exploding cakes" from public recipe books (Kaggle notebooks).

Not just any mistakes: They specifically looked for crashes that stop the program dead in its tracks.
The "Before and After": For every single crash, they didn't just leave it broken. They manually fixed it. So, for every 111 broken notebooks, there is now a matching "Fixed" version.
The Ingredients: They made sure the crashes came from all the popular tools data scientists use, like TensorFlow, PyTorch, and Scikit-learn, as well as mistakes specific to the notebook style (like running a step before the ingredients were even mixed).

2. The Lab (The "Reproducible Kitchen")

One of the biggest headaches in fixing computer bugs is that a crash might happen on one computer but not another because of different software versions.

The Solution: JunoBench comes with a Docker image. Imagine this as a sealed, self-contained kitchen box. No matter who opens it, the oven, the mixer, and the ingredients are exactly the same. This ensures that if a crash happens in the box, it will happen every single time for every researcher. This makes testing fair and reliable.

3. The Labels (The "Diagnosis Cards")

You can't just say "it broke." You need to know how it broke. The researchers acted like expert doctors and gave every crash a detailed medical report:

Who caused it? (Was it the TensorFlow library? Or the Pandas data tool?)
What was the symptom? (Did the numbers get the wrong shape? Did a variable disappear?)
Why did it happen? (Did the user type the wrong command? Did they forget to load the data first?)
Where in the process? (Did it happen while building the model, training it, or looking at the results?)

4. Why This Matters (The "Training Ground")

Before JunoBench, if a researcher wanted to build a tool to automatically fix these crashes, they had to guess or use messy, inconsistent examples.

The New Standard: Now, researchers can take their new "fix-it" tools and run them against JunoBench. They can see: "Did your tool find the crash? Did it know which library caused it? Did it fix it correctly?"
Specific Challenges: The paper highlights that notebook crashes are tricky because the computer "remembers" things from previous steps (like a variable defined in cell #1 being used in cell #10). JunoBench helps test if new tools can understand this "memory" and the order of events.

What JunoBench is NOT (Based strictly on the paper)

It is not a tool to fix your own code right now.
It is not a collection of every possible bug in the world; it's a curated, balanced sample of 111 specific ones.
It does not include "silent" bugs where the code runs but gives the wrong answer; it only includes "loud" crashes that stop the program.

In a nutshell: JunoBench is a standardized, reproducible, and well-labeled collection of 111 broken AI notebooks and their fixes. It is designed to be the "final exam" for new software tools that promise to help developers find and fix these specific types of errors faster and better.

1. Problem Statement

Jupyter notebooks are the dominant environment for prototyping and experimenting with Machine Learning (ML) and Deep Learning (DL) in Python. However, debugging support for notebook-based ML development is limited. This limitation stems from a lack of realistic, executable benchmark datasets specifically designed for this environment.

Existing bug benchmarks for ML (e.g., Defect4ML, gDefects4DL) primarily focus on:

Script-based artifacts (.py files): They ignore the unique semantics of notebooks (.ipynb), such as persistent execution state and non-linear (out-of-order) cell execution.
Heterogeneous symptoms: They often mix crashes with performance degradation or incorrect functionality, making it difficult to evaluate crash-specific debugging tools.
Reproducibility issues: Many lack standardized execution environments, making it hard to reliably reproduce bugs across different systems.

The authors argue that crashes (program terminations) are the most disruptive and observable symptom in ML programs, making them the ideal target for evaluating automated debugging, diagnosis, and repair techniques.

2. Methodology

The construction of JunoBench followed a rigorous, multi-stage pipeline (illustrated in Figure 1 of the paper) to ensure reproducibility, balance, and validity.

A. Data Collection & Filtering

Source: Derived from a prior empirical study analyzing 64,031 notebooks (GitHub and Kaggle).
Selection Criteria:
1. Platform: Restricted to Kaggle notebooks to ensure access to public datasets and a controlled environment.
2. Relevance: Focused on crashes related to ML libraries (e.g., TensorFlow, PyTorch) or notebook-specific issues (e.g., out-of-order execution).
3. Exclusion: Removed hardware-dependent crashes (e.g., GPU memory limits) or environment-specific bugs that cannot be deterministically reproduced.
Resampling: To address uneven distribution across libraries, the authors resampled from an unlabeled pool of 3,519 crashes. They used a heuristic tool to prioritize notebooks based on library imports and traceback analysis, aiming for balanced coverage.

B. Benchmark Inclusion & Reproduction

Isolation: Each benchmark instance is an independent notebook containing a single, isolated crash.
Three Versions per Instance:
1. Original: As collected.
2. Reproduced: Minimal adjustments (e.g., execution order, paths) to ensure the crash occurs exactly as in the original.
3. Fixed: A manually repaired version where the crash is resolved while preserving the original intent.
Data Handling: Input datasets were downsampled (e.g., reducing a 2.5GB image dataset to 94MB) if the crash behavior was preserved, to reduce storage and execution overhead. Synthetic data was used for restrictive licenses.

C. Labeling & Annotation

Categorization: Using grounded theory, annotators labeled each crash across four dimensions:
1. Library Cause: Which library triggered the crash (e.g., NumPy, Pandas, PyTorch).
2. Crash Type: Specific error manifestation (e.g., tensor shape mismatch, variable not found).
3. Root Cause: The underlying developer error (e.g., API misuse, data confusion).
4. ML Pipeline Stage: Where the crash occurred (e.g., data prep, training, evaluation).
Diagnosis Labels: Natural language summaries explaining the failure mechanism and localization, validated by a second annotator.
Quality Control: Inter-annotator agreement was high (Cohen's $\kappa > 0.9$ ).

D. Execution Environment

Unified Docker Image: JunoBench provides a single Docker image containing all necessary dependencies to ensure crashes reproduce consistently across different machines.
CLI Tool: An automated command-line tool verifies that the "reproduced" version crashes and the "fixed" version runs successfully.

3. Key Contributions

First Executable Benchmark for ML Notebooks: JunoBench is the first dataset providing 111 curated, reproducible real-world crashes specifically from Python ML Jupyter notebooks, paired with verified fixes.
Comprehensive Coverage:
- Libraries: Covers major DL/ML libraries (TensorFlow/Keras, PyTorch, Scikit-learn) and data science libraries (Pandas, NumPy, Matplotlib, Seaborn).
- Notebook Semantics: Includes 20 crashes specifically caused by out-of-order execution and state inconsistencies, a unique challenge in notebooks.
Structured Ground Truth: Each crash includes human-validated labels for library cause, crash type, root cause, pipeline stage, and natural language diagnostic summaries.
Reproducibility Infrastructure: A unified Docker environment and CLI tool that allow researchers to reproduce all 111 cases with a single command, eliminating environment configuration barriers.

4. Results & Dataset Characteristics

The final dataset comprises 111 reproducible crashes with the following distribution:

Library Cause: Balanced distribution with ~15 crashes each for TensorFlow/Keras, PyTorch, Scikit-learn, NumPy, and Pandas. Includes 20 notebook-specific crashes and 16 involving visualization or minor libraries.
Root Cause:
- API Misuse: 45 cases (most frequent, e.g., invalid arguments).
- Data Confusion: 25 cases (shape/type mismatches).
- Implementation Errors: 18 cases (logic flaws).
- Notebook-specific: 20 cases (state issues).
Crash Types: Includes invalid arguments (21), tensor shape mismatches (16), variable not found (10), and unsupported broadcasts (5).
Pipeline Stage: Crashes occur across the entire ML lifecycle, with the highest frequency in Model Training (31) and Data Preparation (30).

5. Significance & Research Opportunities

JunoBench enables systematic evaluation of debugging techniques that were previously impossible to assess due to the lack of data. Key research directions it supports include:

Crash Detection & Automated Repair: Evaluating tools that can identify crashes across multiple cells and suggest fixes that account for persistent kernel state.
Pipeline-Aware Debugging: Studying how crashes propagate from data preparation stages to model training, requiring cross-library understanding.
LLM-Based Analysis: Providing a testbed for Large Language Models (LLMs) to reason about stateful execution and cross-cell dependencies. The paper notes that current LLMs struggle with this, achieving F1-scores between 57.7% and 64.6%.
Agent-Based Debugging: Supporting the development of autonomous agents that can iteratively inspect variables, re-execute cells, and propose fixes within a notebook environment.

Conclusion

JunoBench fills a critical gap in ML software engineering research by providing the first standardized, executable benchmark for notebook crashes. By combining realistic data, verified fixes, structured annotations, and a reproducible environment, it serves as a foundational resource for advancing automated debugging, diagnosis, and repair tools for the interactive ML development workflow.