🤖 machine learning

Automated Modernization of Machine Learning Engineering Notebooks for Reproducibility

This paper introduces MLEModernizer, an LLM-driven agentic framework that automatically modernizes machine learning engineering notebooks to overcome environmental erosion and restore reproducibility by iteratively executing code and applying targeted fixes, successfully recovering over 40% of previously non-reproducible notebooks.

Original authors: Bihui Jin, Kaiyuan Wang, Pengyu Nie

Published 2026-07-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bihui Jin, Kaiyuan Wang, Pengyu Nie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a time traveler who loves to look at old recipes. You find a famous cake recipe from 2015 written in a special notebook. The recipe says, "Mix flour, sugar, and a magical ingredient called 'XGBoost' to get a perfect cake." You try to bake it today, but something goes wrong. The "XGBoost" ingredient you buy at the store today is slightly different; it's stronger, it works differently, and when you mix it in, the batter explodes or tastes like soap. This is the problem of "environmental erosion." In the world of Machine Learning Engineering (MLE), scientists and engineers use interactive notebooks (like digital recipe books) to build AI models. They share these notebooks so others can learn from them or use their code. But as computers and software update rapidly, old notebooks often break because the "ingredients" (software libraries) they rely on have changed or vanished. If you can't run the old code, you can't verify the results, and the knowledge gets lost. The big question is: Can we fix these old, broken recipes so they work in today's kitchen without having to rebuild the entire kitchen from scratch?

This paper, titled "Automated Modernization of Machine Learning Engineering Notebooks for Reproducibility," dives into this exact problem. The authors, Bihui Jin, Kaiyuan Wang, and Pengyu Nie, decided to test just how broken these digital recipes really are. They gathered a massive collection of 12,106 notebooks from popular machine learning competitions on Kaggle. When they tried to run these notebooks in a modern environment, they found a harsh reality: only 26% of them actually worked and produced the same results as they did years ago. The other 74% were broken, either crashing with errors or producing completely different scores.

You might think the solution is simple: just go back in time and install the exact old software versions the recipe was written for. This is called "environment backporting." The authors tried this by downgrading all the software to match the date the notebook was originally submitted. Surprisingly, this made things worse! Instead of 26% working, only 12% became reproducible. It turns out that trying to reconstruct the past is a trap; the old software versions are often missing, incompatible with modern hardware, or just too fragile to run. The authors argue that instead of trying to rebuild the old kitchen, we should fix the recipe itself to work in the new kitchen.

To solve this, the team built a tool called MLEModernizer. Think of this as a super-smart, robotic sous-chef powered by a Large Language Model (LLM). This robot doesn't just read the recipe; it actually tries to bake the cake. If the cake burns (an error occurs), the robot looks at the smoke, figures out what went wrong, and rewrites the recipe to fix it. If the cake takes too long to bake (runtime issues), the robot tweaks the instructions to speed it up. If the cake tastes slightly different (the score is off), the robot adjusts the ingredients to get the taste back to the original standard.

The robot works in a loop: it runs the code, checks the results, and if something is wrong, it asks the AI to write a patch (a fix) for the whole notebook. It keeps doing this up to 16 times until the notebook works. The results were quite promising. When they tested this on 8,210 broken notebooks, the tool managed to fix about 40% to 45% of them, making them reproducible again. Specifically, using a powerful AI model (GPT-5.2), they fixed 3,292 notebooks, and with a slightly different open-source model (GPT-OSS-120b), they fixed 3,683.

However, the paper is careful not to call this a magic wand. The authors found that while the tool could get the numbers right (the "score"), sometimes the underlying code changed in ways that weren't exactly the same as the original. For instance, the AI might change how a model is trained just enough to get the right score, but the "flavor" of the code is different. They also discovered that some errors are just too stubborn to fix automatically; the tool was great at fixing simple "missing ingredient" errors (like forgetting to import a library), but it struggled with complex "weird ingredient interaction" errors (like attribute errors where an object doesn't have the right properties).

The cost of this robotic chef is also a factor. The authors calculated that fixing one notebook costs about $0.65 on average using the powerful model, which they suggest is cheap enough for large-scale use. But they also noted that the open-source model was much cheaper (about $0.0074 per notebook) but didn't keep the code as faithful to the original.

In the end, the paper suggests that while we can't perfectly recreate the past, we can use smart AI agents to modernize old code and save valuable scientific work. It's not a perfect fix for every single broken notebook, but it offers a "best-effort" way to rescue a significant chunk of machine learning history that would otherwise be lost to time. The authors conclude that this approach helps practitioners validate and reuse old pipelines as technology continues to evolve, turning broken, dusty recipes into something that can be cooked in today's kitchen.

Technical Summary: Automated Modernization of Machine Learning Engineering Notebooks for Reproducibility

1. Problem Statement

Interactive computational notebooks (e.g., Jupyter) are the primary artifacts for documenting and sharing end-to-end Machine Learning Engineering (MLE) pipelines. However, these notebooks suffer from environmental erosion: the rapid evolution of hardware and software ecosystems renders many published notebooks non-reproducible in contemporary environments.

The authors identify a critical gap in the current state of reproducibility:

The Gap: A large-scale study of 12,106 notebooks from 75 Kaggle competitions revealed that only 26% remain reproducible in a modern environment.
The Failure of Backporting: The standard industry practice of "environment backporting" (downgrading dependencies to match the notebook's submission timestamp) was tested and found to be counterproductive. Instead of improving reproducibility, backporting reduced the success rate to 12% and introduced additional failure modes (e.g., dependency conflicts, unavailable historical wheels).
The Core Challenge: Simply running code in a reconstructed historical environment is often infeasible or insufficient. The code itself must be modernized to run and reproduce scores in the current environment. Crucially, "reproducibility" in MLE is not just about execution (no errors); it requires score-calibration, where the reproduced model performance remains within an acceptable band of the originally reported score.

2. Methodology: MLEModernizer

To address environmental erosion, the authors designed and implemented MLEModernizer, an LLM-driven agentic framework. Unlike traditional repair tools that treat the environment as a variable to be reconstructed, MLEModernizer treats the contemporary environment as a fixed constraint and iteratively modernizes the notebook code.

Agentic Workflow

The framework operates in an iterative loop with a maximum of 16 iterations per notebook:

Execution & Feedback: The notebook is executed in a containerized environment (aligned with Kaggle's execution model). The system collects execution feedback, including:
- Runtime duration.
- Error tracebacks (if any).
- Reproduced score ( $s_r$ ) compared to the target score ( $s_t$ ).
Issue Detection: The system classifies the failure into one of three categories:
- Timeout: Execution exceeds the 600-second limit.
- Error: The notebook raises exceptions.
- Score Deviation: The notebook runs without errors but the score deviates significantly from the target.
Targeted LLM Fixing: Based on the detected issue, MLEModernizer constructs a specific prompt for a Large Language Model (LLM) to generate a patch. The framework supports three distinct fix types:
- Error-Repair: Fixes execution errors (e.g., API breaking changes, missing imports).
- Runtime-Reduction: Optimizes code to prevent timeouts (e.g., enabling GPU acceleration, reducing data subsets).
- Score-Calibration: Adjusts the code to align the reproduced score with the target (e.g., handling label shifts, hyperparameter tuning) when no errors exist.
Patch Application: The LLM returns a plan and a full-file patch (the entire notebook). MLEModernizer applies this patch and repeats the loop until the notebook is reproducible or the iteration limit is reached.

Evaluation Setup

Dataset: 8,210 non-reproducible notebooks (filtered from the initial 12,106) were selected for modernization.
Models: The system was evaluated using GPT-5.2 (proprietary) and GPT-OSS-120b (open-weight).
Metrics: Reproducibility was defined by a one-sample $t$ -test ( $p \ge 0.05$ ) or a heuristic threshold ( $\tau = 10\%$ ) if scores were deterministic. Functional equivalence was assessed via manual inspection of a statistically sampled subset.

3. Key Results

Reproducibility Recovery

Success Rates: MLEModernizer successfully restored reproducibility to 40.1% of the non-reproducible notebooks using GPT-5.2 (3,292 notebooks) and 44.9% using GPT-OSS-120b (3,683 notebooks).
Error-Free vs. Error-Containing:
- With GPT-5.2, 2,397 notebooks became "Error-Free Reproducible," while 895 were "Error Reproducible" (reproducible despite residual errors).
- With GPT-OSS-120b, 2,877 became "Error-Free Reproducible," and 806 were "Error Reproducible."
Functional Equivalence: While GPT-OSS-120b achieved a higher raw reproducibility rate, GPT-5.2 produced significantly more functionally equivalent modernizations (71.5% vs. 52.7%). This suggests a trade-off where the open model achieves scores more aggressively, potentially altering semantics, while the proprietary model better preserves the original logic.

Fix Statistics and Costs

Iteration Efficiency: The majority of successful modernizations required few fixes (median of 2). However, achieving "Error-Free" status typically required more iterations (mean 4.35) than achieving "Error Reproducible" status (mean 2.56).
Fix Types: Score-Calibration was the most frequent fix type applied (39,226 times), followed by Error-Repair (27,765) and Runtime-Reduction (15,718). Score-Calibration was the primary driver for "Error-Free Reproducible" outcomes.
Cost: The process is economically viable for large-scale use.
- GPT-5.2: Average cost of $0.65 USD per notebook.
- GPT-OSS-120b: Average cost of $0.0074 USD per notebook.
Code Modification: While individual fixes are often incremental (high edit similarity to the previous version), cumulative modifications can be substantial. Notebooks that initially timed out underwent the most significant code divergence from the baseline.

Error Analysis

Error Reduction: The system reduced total error counts from >92,000 at baseline to ~4,000 after 16 fixes.
Error Types: NameError (undefined names/missing imports) was effectively reduced. However, AttributeError (missing/invalid attributes) proved the most stubborn, remaining the dominant error type in both reproducible and non-reproducible notebooks after modernization. This indicates that API drift and library-specific attribute changes are harder to automate than simple syntax or import fixes.

4. Significance and Claims

The paper positions MLEModernizer as a best-effort automated recovery and modernization technique. The authors make the following claims regarding its significance:

Paradigm Shift: The work challenges the prevailing paradigm of "environment reconstruction" (backporting). It demonstrates that treating the environment as a fixed constraint and modernizing the code is a more effective strategy for restoring reproducibility in rapidly evolving ML ecosystems.
Practical Utility: By enabling the automated modernization of MLE notebooks, the tool facilitates the validation and reuse of legacy ML pipelines. This is critical for both researchers (transferring knowledge from legacy prototypes) and engineers (maintaining long-term usability of production pipelines).
Limitations and Scope: The authors are modest about the tool's capabilities. They acknowledge that:
- Exact reproduction may be impossible due to evolved environments; the tool provides a starting point for manual refinement.
- Score-based reproducibility does not guarantee functional equivalence (though the tool attempts to preserve it).
- Certain error types (specifically complex AttributeErrors) remain difficult to resolve automatically.
- The evaluation relies on offline grading (MLE-Bench) which approximates but does not perfectly replicate Kaggle's private leaderboard.

In conclusion, the paper asserts that while code modernization is not a silver bullet, it offers a viable, scalable path to restoring reproducibility for a significant subset of legacy MLE artifacts, bridging the gap between historical code and contemporary infrastructure.