💬 NLP

DEEP: Docker-based Execution and Evaluation Platform

The paper introduces DEEP, a Docker-based platform that automates the execution, scoring, and statistical clustering analysis of machine translation and optical character recognition models, while providing a web interface for visualizing and interpreting performance differences among competing systems.

Original authors: Sergio Gómez González, Miguel Domingo, Francisco Casacuberta

Published 2026-02-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sergio Gómez González, Miguel Domingo, Francisco Casacuberta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a massive cooking competition. You have 8 different chefs (the AI models), and your job is to figure out who makes the best dish. But there's a catch: you don't just want to taste the food; you also want to know how long it took them to cook, how much electricity their ovens used, and whether their "better" dish is actually significantly better or just a lucky fluke.

In the past, judges had to ask chefs to send in their recipes (the code) and their finished dishes (the predictions). But then, the judges couldn't see how the chefs cooked. Did they use a tiny microwave or a giant industrial oven? How long did it take?

DEEP (Docker-based Execution and Evaluation Platform) is like a high-tech, automated kitchen built specifically to solve this problem. Here is how it works, broken down into simple steps:

1. The "Magic Box" (Docker Containers)

Instead of asking chefs to send their recipes, DEEP asks them to bring their entire kitchen in a sealed, portable box (called a Docker container).

How it works: You give this box a list of ingredients (the test data). The box cooks the meal on its own, and then spits out the finished dish.
Why it's cool: Because the box is sealed, the judges can watch exactly how long the cooking took and how much energy was used, without needing to know the secret recipe inside. It's like watching a chef cook through a glass wall without being able to touch the stove.

2. The "Taste Test" (Evaluation)

Once the boxes have cooked their meals, DEEP acts as the super-judge. It compares every dish against the "perfect" recipe (the reference).

The Metrics: It uses different rulers to measure the food.
- BLEU & chrF: Like checking if the ingredients are in the right order and if the flavors match the original.
- TER & WER: Like counting how many times the chef had to swap an ingredient or delete a word to make it taste right.
The Twist: DEEP doesn't just say "Chef A is #1." It uses a statistical magic trick (called clustering) to ask: "Is Chef A's dish actually that much better than Chef B's, or are they basically the same?"
- If the difference is tiny, DEEP puts them in the same "team" (cluster).
- If the difference is huge, it puts them in different teams. This prevents you from getting excited over a 0.1% improvement that might just be noise.

3. The "Dashboard" (Visualization)

After the judging is done, DEEP gives you a colorful dashboard (a web app) to look at the results. Think of it like a sports analytics screen.

The Charts:
- Bar Charts: Show who got the highest score.
- Time Charts: Show who cooked the fastest.
- The "Sweet Spot" Chart: This is the most important one. It plots Quality vs. Speed.
  - Some chefs make amazing food but take 10 hours (Heavy Models).
  - Some chefs are super fast but the food is mediocre (Fast but Low Quality).
  - Some chefs are the "Goldilocks" zone—great food, cooked quickly (Balanced Models).

The Real-World Test (The Case Study)

The authors tested DEEP with 8 famous AI "chefs" (like NLLB, M2M-100, and a new one called Seed) to translate English to German.

The Surprise: One new chef, Seed, was the winner. It didn't just make the best translation; it did it faster than almost everyone else. It was the "fastest and best" chef in the room.
The Lesson: Without DEEP's ability to measure both speed and quality simultaneously, you might have just picked the "best tasting" dish and ignored the fact that it took 10 hours to make, while missing the chef who was both fast and delicious.

Why Should You Care?

DEEP is like a universal translator for AI performance. Whether you are translating text (Machine Translation) or reading handwriting from images (OCR), this tool lets researchers and companies:

Stop guessing which AI is best.
See the trade-offs (Speed vs. Quality).
Know for sure if a new model is actually an improvement or just a statistical fluke.

In short, DEEP takes the chaos of comparing AI models and turns it into a clear, fair, and easy-to-read scoreboard.

1. Problem Statement

In machine learning research and competitive challenges (such as those for Machine Translation or Optical Character Recognition), evaluating multiple systems presents several challenges:

Resource Opacity: Traditional methods where participants submit only hypotheses prevent evaluators from measuring the computational resources (time, hardware) required to generate those results.
Test Set Leakage: Conversely, methods where participants submit full systems often require sharing test sets, risking data leakage.
Lack of Holistic Analysis: Standard evaluation often relies on raw metric scores without statistical significance testing to determine if performance differences are meaningful. Furthermore, there is a lack of tools to visualize the trade-off between model quality and execution efficiency.
Reproducibility: Ensuring that different systems run in identical environments to guarantee fair comparison is difficult without containerization.

2. Methodology

The authors propose DEEP (Docker-based Execution and Evaluation Platform), an automated, modular pipeline designed to execute, evaluate, and visualize the performance of ML models. The system operates in three distinct phases:

A. Execution Phase (Docker-based)

Containerization: Systems must be packaged as Docker images. This ensures a consistent runtime environment, isolating dependencies and preventing "it works on my machine" issues.
Data Handling: The platform accepts source data (text files for Machine Translation, image directories for OCR) and feeds them into the Docker containers.
Monitoring: During execution, DEEP captures metadata, specifically the computation time required to generate hypotheses.
Cleanup: Upon completion, Docker images and temporary files are removed to manage disk usage.

B. Evaluation Phase

Metric Calculation: The platform computes scores based on user-selected metrics.
- For Machine Translation (MT): BLEU, TER, chrF, and BEER.
- For Optical Character Recognition (OCR): Word Error Rate (WER) and Bag-of-Words WER (bWER).
- Extensibility: The architecture allows users to plug in custom metrics provided they accept hypotheses and references as input and return a float score.
Statistical Clustering: To address the issue of whether score differences are meaningful, DEEP employs Approximate Randomization Testing (ART).
- Systems are sorted by their primary metric score.
- ART (with 10,000 trials and $p=0.05$ ) compares adjacent systems.
- If the difference is not statistically significant, systems are grouped into the same performance cluster. This helps identify groups of models that perform equivalently rather than just ranking them linearly.

C. Visualization Phase

Web Interface: Built using Streamlit, the platform provides an interactive dashboard.
Visualizations:
- Bar Charts: Display scores ordered from best to worst, with baselines highlighted.
- Arc Charts: Visualize computation time per system.
- Scatter Plots: Plot Quality (Score) vs. Time (Efficiency) to identify optimal trade-offs.
- Cluster Charts: Show average scores per statistical cluster.
Data Export: Results can be exported to CSV, LaTeX, JSON, or HTML.

3. Key Contributions

Unified Pipeline: DEEP integrates execution, evaluation, and visualization into a single, open-source framework, specifically designed for MT and OCR but extensible to other tasks.
Resource-Aware Evaluation: Unlike traditional evaluation, DEEP captures execution time, enabling a holistic view of model efficiency alongside accuracy.
Statistical Rigor: The implementation of ART-based clustering moves beyond simple ranking, allowing researchers to identify statistically significant performance tiers among a "swarm" of proposals.
Extensibility: The modular design allows for easy adaptation to new tasks (e.g., summarization) and the addition of new metrics (including potential LLM-based metrics) without rewriting the core engine.
Reproducibility: By enforcing Docker-based execution, DEEP guarantees that all models are evaluated under identical environmental constraints.

4. Results (Case Study)

The authors validated DEEP using a case study involving 8 state-of-the-art open-source models (including EuroLLM, M2M-100, MADLAD-400, Seed, NLLB-200, etc.) on the WMT20 English-to-German test set (1,418 sentences).

Performance Ranking: The tool successfully identified Seed, MADLAD-400, and NLLB-200 as the top performers based on BLEU and chrF scores.
Efficiency Analysis:
- Opus-MT and mBART were identified as the fastest models.
- Seed was highlighted as a "Balanced" model, offering the best performance while maintaining high speed (236s), outperforming heavier models like MADLAD-400 (870s).
Clustering Insights:
- For BLEU and chrF, every model fell into its own unique cluster (all differences were significant).
- For the TER metric, Seed and MADLAD-400 fell into the same cluster, indicating their performance difference was not statistically significant despite the raw score gap.
Visualization Utility: The scatter plots effectively separated models into "Balanced," "Heavy" (high quality, high time), and "Improvement Zone" (low performance, high time) categories, providing immediate strategic insights for researchers.

5. Significance

DEEP addresses a critical gap in the ML evaluation lifecycle by shifting from static, score-only reporting to dynamic, resource-aware, and statistically robust analysis.

For Competitions: It offers a fair, transparent, and automated way to judge submissions without exposing test data or hiding resource costs.
For Researchers: It facilitates the identification of the "Pareto frontier" of models—those that offer the best balance between accuracy and computational cost.
Future Impact: By being open-source and modular, DEEP sets a standard for reproducible evaluation, encouraging the community to adopt containerized execution and statistical significance testing as standard practices in NLP and computer vision research.