Technical Summary: ML in a Box

Problem Statement

Containerization has become the standard for building, testing, and deploying machine learning (ML) systems, offering reproducibility and portability. However, ML workflows are inherently iterative and experimental, involving frequent changes to code, configurations, models, and data. This rapid iteration cycle interacts poorly with Docker's linear, layered build mechanics. While prior research has analyzed general Dockerfile structures and "smells," there is a lack of systematic, role-aware empirical evidence regarding:

1. The specific functional roles containers serve within ML projects.
2. The actual footprint and cold-build latency of ML containers in the wild.
3. The frequency of rebuilds, the specific triggers (Dockerfile edits vs. context file changes), and the efficiency of cache reuse once a rebuild is triggered.
4. The specific refactoring patterns practitioners employ to mitigate these inefficiencies.

Existing studies often treat all Dockerfiles in ML repositories as homogeneous or focus on general ecosystem trends without linking image size, rebuild mechanics, and developer intent to concrete ML-specific practices.

Methodology

The authors conducted a large-scale empirical study analyzing 1,993 ML-related Dockerfiles from 392 open-source Python-based ML projects. The study design addressed three Research Questions (RQs) through a multi-stage process:

• Data Collection: Starting from a corpus of 31,066 Python ML repositories, the authors filtered for public, actively maintained projects with end-to-end ML pipelines. They collected Dockerfiles, validated them syntactically, and used an LLM (GPT-4o) to filter for ML-specific containers, followed by manual verification.
• RQ1 (Roles, Size, Latency): The 1,993 Dockerfiles were manually classified into five functional roles: Training, Inference, Sandbox, Infrastructure, and Template. Successful builds were executed on a standardized workstation (AMD Ryzen 7, 32GB RAM, NVIDIA RTX 2060) to measure image size and wall-clock build time. Failed builds were also analyzed for partial size and time to failure.
• RQ2 (Rebuild Dynamics): The authors analyzed 1.06 million commits across 939 Training and Inference Dockerfiles. They identified rebuild-triggering commits (changes to the Dockerfile or copied context files). To estimate cache efficiency without executing historical builds, they developed a static heuristic based on Docker's cache semantics. This heuristic calculates Cache-Break Depth (CBD) (how far into the build the cache breaks) and Cache Reuse Share (CRS) (the proportion of work reused). Commit intents were labeled using an LLM (GPT-4o) into six categories (e.g., Experiment, Bug Fix, Dependency/Env).
• RQ3 (Refactoring Patterns): From projects exhibiting high cache reuse (top 25th percentile), the authors extracted 377 Dockerfile-modifying commits. Three authors manually reviewed these to identify recurring, behavior-preserving refactoring patterns specific to ML workflows.

Key Results

1. Container Roles and Footprint (RQ1)

• Roles: Containers are distributed as: Inference (31.7%), Sandbox (29.2%), Training (15.4%), Infrastructure (14.2%), and Templates (9.5%).
• Size: ML containers are typically large. The overall median size is 10.27 GB, with Training containers being the largest (median 17.25 GB, max 125 GB). Inference and Infrastructure containers are significantly smaller (medians ~1.5–1.7 GB).
• Build Time: Build times correlate with size. Training containers have a median build time of 14.64 minutes, while Inference and Infrastructure take roughly 1 minute.
• Layer Composition: Base images contribute only 16–21% of the total size. The majority (79–84%) comes from custom layers, dominated by RUN instructions (dependency installation) and COPY/ADD instructions (code, models, data).
• Failure Overhead: 59% of builds in the dataset failed. Even failed builds accumulate significant intermediate data (median 2.16 GB) and time (median 4.56 min for Training), creating a slow feedback loop.

2. Rebuild Frequency and Cache Efficiency (RQ2)

• Frequency: 44.4% of all commits trigger a rebuild. Training containers are rebuilt more frequently per Dockerfile (median 19.5%) compared to Inference (10.4%).
• Triggers: The primary cause of rebuilds is context file changes (copied files), accounting for 96.4% of triggers. Dockerfile-only edits are rare (1.1%).
• Cache Inefficiency: Once a cache break occurs, reuse is low. The median Cache Reuse Share (CRS) is 0.286, meaning approximately 71% of the build work is wasted (re-executed) even when only a small file changes.
• Intent Analysis:
  • Training rebuilds are driven primarily by Experiment commits (46%), reflecting iterative model tuning.
  • Inference rebuilds are driven by Feature/Enhancement (18%), Infrastructure/CI (17%), and Dependency/Environment (13%) changes.
  • Dependency/Environment and Infrastructure/CI changes are particularly damaging, often triggering near-cold rebuilds (CRS < 0.3) because they frequently invalidate early, heavyweight dependency layers.

3. ML-Specific Refactoring Patterns (RQ3)

The study identified 7 recurring refactoring patterns used to improve build efficiency and reduce footprint:

1. Externalize Dataset Artifact: Removing dataset COPY steps from the Dockerfile and mounting data at runtime.
2. Externalize Model Weights: Removing model file copies and loading weights at runtime (bind-mount or download).
3. Reallocate Model Downloads: Moving deterministic model downloads to the beginning of the Dockerfile to cache them across code changes.
4. Split Container Image by Accelerator: Separating CPU and GPU targets into distinct Dockerfiles to avoid bundling unnecessary stacks.
5. Select Hardware-Targeted ML Framework: Installing only the specific framework variant (e.g., CPU-only PyTorch) to avoid pulling unused binaries.
6. Reallocate Model Configuration: Copying frequently changing configuration files after heavy dependency installations to prevent invalidating the cache.
7. Shallow-Clone VCS Dependencies: Using git clone --depth 1 instead of full history clones to reduce network and unpack time.

Significance and Claims

The paper claims to provide the first large-scale empirical study linking ML development activities to Docker build times, image footprint, and cache fragility. Its significance lies in:

• Establishing a Baseline: Quantifying the "cost" of ML containers (size, time, rebuild frequency) and revealing that standard Docker practices often lead to significant waste (70% of work re-executed) in iterative ML workflows.
• Role-Aware Insights: Demonstrating that different container roles (Training vs. Inference) have distinct size profiles and rebuild drivers, necessitating different optimization strategies.
• Actionable Patterns: Curating a catalog of 7 ML-specific refactoring patterns grounded in real-world commits that developers can adopt to reduce image size and build times.
• Methodological Contribution: Introducing a static heuristic (CRS/RWS) to estimate rebuild costs without executing historical builds, enabling analysis at the scale of millions of commits.

The authors conclude that ML containerization must evolve from ad-hoc reuse of general Docker practices toward ML-specific discipline, focusing on externalizing large artifacts, reordering build steps to protect cache, and separating hardware targets. These insights call for ML-aware CI/CD tooling and static analyzers to make experimentation faster and more sustainable.