💻 computer science

Runtime-Augmented LLMs for Crash Detection and Diagnosis in ML Notebooks

The paper introduces CRANE-LLM, a novel approach that enhances large language models with structured runtime information from notebook kernels to significantly improve the accuracy and F1-score of crash detection and diagnosis in machine learning notebooks before code execution.

Original authors: Yiran Wang, José Antonio Hernández López, Ulf Nilsson, Dániel Varró

Published 2026-02-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiran Wang, José Antonio Hernández López, Ulf Nilsson, Dániel Varró

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to cook a complex meal in a kitchen where the rules keep changing. You have a recipe (the code), but you also have a pantry full of ingredients that might have spoiled, changed size, or been swapped out without you noticing (the runtime state).

In the world of Machine Learning (ML), developers use "Jupyter Notebooks" as their kitchen. It's a place where they write code in small chunks (cells) and run them one by one to see what happens. The problem? If you make a mistake in the middle of cooking, the whole kitchen can get messy. You might burn a pot, spill flour everywhere, and the computer (the "kernel") gets stuck in a confused state. To fix it, you often have to throw away the whole kitchen, clean it, and start cooking from the very first step again. This is a huge waste of time.

This paper introduces CRANE-LLM, a new "smart sous-chef" designed to stop these disasters before they happen.

The Problem: The "Silent Kitchen Disaster"

Usually, when you write code, you just look at the recipe (the text) to find errors. But in ML, the recipe often depends on things you can't see just by reading.

Example: Your recipe says, "Mix 5 cups of flour." But in your actual pantry, you only have 2 cups left because you used the rest earlier. If you just read the recipe, you think everything is fine. But when you try to mix, the bowl explodes (the code crashes).
The Consequence: In a notebook, if the bowl explodes, the kitchen is ruined. You can't just fix the bowl; you have to restart the whole cooking session.

The Solution: CRANE-LLM (The Smart Sous-Chef)

The researchers built a system that doesn't just read the recipe; it looks into the pantry before you start cooking the next step.

The "Pantry Check" (Runtime Information Extraction):
Before you run a new cell of code, CRANE-LLM pauses and asks the computer: "Hey, what do we actually have right now?"
It checks the "state" of the kitchen:
- How big is the data table? (Is it 10 rows or 1,000?)
- What shape are the ingredients? (Are we trying to fit a square peg in a round hole?)
- Are there any spoiled ingredients? (Missing values or weird data types?)
The "Brain" (The Large Language Model):
It takes this "Pantry Check" report and combines it with the recipe (the code). It then asks a super-smart AI (like GPT-5 or Gemini) to act as a detective.
- The Question: "Based on the recipe and the actual ingredients in the pantry, will this next step crash?"
- The Answer: The AI says, "Yes! You're trying to feed a 2-cup ingredient into a 5-cup machine. Stop now!"

Why This is a Big Deal

The paper tested this system on hundreds of real-world cooking disasters (bugs) using three different "smart chefs" (AI models). Here is what they found:

It Works Better with Eyes Open: When the AI was allowed to look at the "Pantry Check" (runtime info), it got significantly better at spotting errors. It improved its accuracy by about 7–10%.
The "Why" Matters More: The AI was even better at explaining why the crash would happen when it had the pantry info. It's like the difference between saying "The car won't start" (detection) and "The car won't start because the gas tank is empty" (diagnosis). The extra info helps the AI give a better explanation.
Not All Info is Equal: The researchers tried giving the AI different types of info:
- Structure: (How big is the table?)
- Type: (Is this a number or a word?)
- Values: (Are there missing numbers?)
  They found that the AI needed a mix of all three to work best, though different AI models liked different types of info.
The "Instruction Manual" Trap: The researchers also tried giving the AI the official instruction manuals (API documentation) for the tools. Surprisingly, this didn't help and actually made things slower and more expensive. The AI already knew the rules; it just needed to know what was actually in the pantry.

The Bottom Line

CRANE-LLM is like a safety net for machine learning developers. Instead of waiting for the kitchen to burn down and then spending an hour cleaning it up, this tool looks at the ingredients and the recipe before you turn on the stove. It tells you, "Don't do that, you'll crash!" saving you time, frustration, and the headache of restarting your entire project.

It turns the process from "Break it, fix it, restart" to "Check it, fix it, keep going."

1. Problem Statement

Machine Learning (ML) development frequently relies on Jupyter Notebooks, which allow for interactive, cell-based execution with a persistent kernel state. While flexible, this environment is prone to specific types of bugs, particularly crashes that halt execution.

The Challenge: Unlike standard Python scripts, notebook crashes can cause irreversible kernel state corruption. If a cell crashes after partially updating variables (e.g., model parameters), the kernel is left in an inconsistent state. Restoring correctness requires restarting the kernel and re-executing all prior cells, which is time-consuming and costly.
Limitations of Existing Solutions:
- Static Analysis: Often fails to detect bugs dependent on runtime data (e.g., tensor shapes, data values) which are unknown until execution.
- Dynamic Analysis: Can only detect errors after they occur, failing to prevent the kernel corruption described above.
- LLMs: While Large Language Models (LLMs) show promise in code analysis, they typically rely on static code context alone, missing critical runtime state information required to predict data-dependent crashes.

2. Methodology: CRANE-LLM

The authors propose CRANE-LLM, a framework that augments LLMs with structured runtime information extracted from the notebook kernel state before executing a target cell.

A. Runtime Information Extraction

The system parses the Abstract Syntax Tree (AST) of the target cell to identify referenced variables and then extracts relevant runtime data from the kernel. To avoid overwhelming the LLM with raw data, the information is summarized into three structured categories:

Structural Information (S): Size, shape, and counts (e.g., tensor shapes, dataset dimensions, sample counts).
Representation and Type Semantics (R): Object types, data types (dtypes), and schema properties (e.g., pandas.DataFrame column types).
Value Semantics (V): Concrete data values and states (e.g., value ranges, presence of NaNs, number of classes, model fit status).

The extraction targets common ML data structures (Tensors, Arrays, DataFrames) and library-specific objects (e.g., Keras ImageDataGenerator, PyTorch DataLoader).

B. LLM-Based Detection and Diagnosis

CRANE-LLM constructs a prompt containing:

The sequence of previously executed cells (static code context).
The target cell to be analyzed.
The extracted, structured runtime information.

The LLM is instructed to perform Chain-of-Thought reasoning to:

Detect: Predict whether the target cell will crash (Boolean output).
Diagnose: Explain the root cause of the potential crash (Natural language reasoning).

The output is enforced via a JSON schema to ensure structured, machine-readable results.

C. Configurations

The framework supports ablation studies by selectively excluding categories of runtime information (e.g., $-S$ , $-R$ , $-V$ ) and includes an optional configuration to ground the LLM with API documentation (though results showed this was counterproductive).

3. Evaluation

Dataset: JunoBench, a benchmark of 222 notebooks (111 buggy, 111 fixed) covering diverse ML libraries (TensorFlow/Keras, PyTorch, Scikit-learn, Pandas, NumPy) and crash root causes.
Models: Evaluated across three state-of-the-art LLMs: Gemini-2.5-Flash, GPT-5, and Qwen-2.5-Coder-32B.
Metrics: Precision, Recall, F1-score, and Accuracy for both crash detection and end-to-end diagnosis.

4. Key Results

The study demonstrates that incorporating runtime information consistently improves LLM performance across all models.

Overall Performance Gains:
- Accuracy: Improved by 7–10 percentage points (pp) across all LLMs.
- F1-Score: Improved by 8–11 pp.
- Diagnosis vs. Detection: The improvement is significantly larger when the task includes diagnosis (+8 to +11 pp) compared to detection alone (+4 to +8 pp). This indicates runtime context is crucial for reasoning about causes, not just identifying errors.
Library-Specific Impact:
- Runtime information provided the most significant gains for PyTorch, Scikit-learn, and Pandas.
- Improvements were mixed for TensorFlow/Keras and NumPy, likely due to the complexity of their abstractions or lower-level operations.
Ablation Study (Runtime Categories):
- Qwen relied heavily on Structural and Representation information.
- GPT-5 was most sensitive to the removal of Value Semantics.
- Gemini showed minor sensitivity to specific categories, suggesting it may rely more on internal code reasoning or other cues.
- Conclusion: A balanced combination of all three categories yields the best results.
API Documentation Grounding:
- Contrary to intuition, adding API documentation decreased performance (by ~1–4 pp) while increasing token costs by ~73% on average. The authors suggest that structured runtime data is more effective than verbose documentation for these specific tasks, and modern LLMs already possess sufficient internal knowledge of common APIs.

5. Key Contributions

Runtime-Augmented Framework: Introduced CRANE-LLM, the first approach to systematically extract and structure runtime kernel state (types, shapes, values) to augment LLMs for pre-execution crash prediction in ML notebooks.
Comprehensive Benchmarking: Validated the approach on JunoBench across multiple libraries and LLMs, proving that runtime context is essential for detecting data-dependent crashes that static analysis misses.
Diagnostic Capability: Demonstrated that runtime information significantly enhances the LLM's ability to explain crashes, not just detect them.
Empirical Insights: Provided evidence that:
- Runtime context is more valuable for diagnosis than simple detection.
- Different LLMs utilize different types of runtime information (structural vs. value-based).
- Adding API documentation is not cost-effective when rich runtime context is already available.

6. Significance

Prevention of Kernel Corruption: By predicting crashes before execution, CRANE-LLM prevents the irreversible corruption of the notebook kernel state, saving developers significant time and effort associated with restarting kernels and re-running expensive preprocessing steps.
Shift in Debugging Paradigm: Moves ML debugging from a reactive process (fixing errors after they crash the kernel) to a proactive one (identifying logical/data mismatches before execution).
Practical Efficiency: The approach adds minimal latency (approx. 1.6 seconds per query) compared to the substantial time saved by avoiding kernel restarts and re-execution of prior cells.
Generalizability: The method generalizes across different ML libraries and even to libraries not explicitly summarized in the extraction logic (e.g., Seaborn), provided they rely on standard data structures like Pandas DataFrames.

In summary, the paper establishes that structured runtime context is a critical missing piece for LLMs in the domain of ML development, and integrating it effectively transforms LLMs into powerful, proactive tools for crash prevention and diagnosis.