💬 NLP

ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation

This paper introduces ML2B, the first benchmark for evaluating cross-lingual machine learning pipeline generation by large language models, which utilizes 490 task-language pairs across 14 languages to reveal that performance degradation is highly task-dependent rather than following traditional resource hierarchies.

Original authors: Ekaterina Trofimova, Zosia Shamina, Maria Selifanova, Artem Zaitsev, Remi Savchuk, Maxim Minets, Daria Ozerova, Emil Sataev, Denis Zuenko, Andrey E. Ustyuzhanin

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ekaterina Trofimova, Zosia Shamina, Maria Selifanova, Artem Zaitsev, Remi Savchuk, Maxim Minets, Daria Ozerova, Emil Sataev, Denis Zuenko, Andrey E. Ustyuzhanin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where you can ask a super-smart robot to build a machine learning model for you, just by typing a simple request like, "Predict which customers will buy our new shoes." For a long time, these robots—called Large Language Models (LLMs)—have been incredibly talented at this, but only if you speak to them in English. It's like having a brilliant chef who can cook a five-star meal if you give the recipe in English, but if you hand them a recipe in French, Spanish, or Japanese, they might just stare at the ingredients and burn the kitchen down.

This is the problem researchers are tackling: as AI spreads across the globe, we need to know if these digital chefs can actually follow instructions in any language, not just the one they were mostly trained on. But there's a catch. If we just ask them to solve puzzles that are already on the internet, the robots might have memorized the answers from their training data, making them look smarter than they really are. It's like giving a student a math test they've already seen the answers to; they get an A, but we don't know if they actually understand algebra. To get a true picture, we need a test that is fresh, fair, and spoken in many different tongues.

Enter ML2B, a new "exam" created by a team of researchers to test how well these AI robots can build complete machine learning pipelines when the instructions are in 14 different languages. Think of ML2B as a massive, multilingual cooking competition. Instead of just asking the robots to write a single line of code, the researchers gave them 35 real-world challenges—like predicting house prices, identifying dog breeds, or spotting toxic comments—and translated the instructions into languages ranging from Arabic and Chinese to Kazakh and Ukrainian.

To make sure the robots couldn't cheat by memorizing answers, the team included 10 "secret" competitions where the solutions were never made public. They also built a special, isolated digital kitchen (a network-isolated container) where the robots had to cook their code. If the robot tried to sneak in a pre-made dish from the internet or used a recipe that mixed up the ingredients before cooking (a problem called "data leakage"), the system would catch them. The robots had to follow strict rules, like separating the "training" phase from the "prediction" phase, just like a real chef wouldn't taste the final dish before it's fully cooked.

When the researchers ran the tests with top-tier AI models, they found something surprising that challenges what many people assumed. They expected that robots would struggle the most with "low-resource" languages (languages with less digital data available) and do best with "high-resource" ones like English or French. But the results showed that the language itself wasn't the main boss. Instead, the difficulty depended entirely on what the robot was being asked to do. Sometimes, a robot did better in a "low-resource" language than in English, and other times it crashed and burned in a "high-resource" one.

The study suggests that the real bottleneck isn't just knowing the words; it's about following the complex, step-by-step instructions of building a machine learning model. The researchers found that the robots often failed not because they didn't understand the language, but because they messed up the structure of the code—like forgetting to close a parenthesis or using a tool that didn't exist in the version of the software they were using. One model, GPT-4.1-mini, was great at coming up with high-quality ideas but often failed to finish the job correctly, while another, GPT-OSS-120b, was more reliable at following the rules but sometimes produced simpler results.

Ultimately, ML2B reveals that we can't just assume an AI is good at everything just because it speaks many languages. The ability to build complex tools depends heavily on the specific task at hand, not just the language used to ask for it. By opening up this new benchmark to the public, the authors hope to help developers build robots that are not only multilingual but also truly robust and reliable, no matter what language they are asked to speak or what problem they are asked to solve.

Technical Summary: ML2B Benchmarking LLMs on Cross-Lingual ML Pipeline Generation

Problem Statement

Current benchmarks for Large Language Model (LLM) code generation, such as MLE-Bench and DA-Code, are limited to English task descriptions. However, machine learning research is global, requiring models to interpret problem descriptions in diverse languages while generating executable code. Existing benchmarks fail to measure cross-lingual robustness in end-to-end ML pipeline generation. Furthermore, many benchmarks suffer from data leakage, where test data or solutions present in the benchmark are also found in the LLM training corpora, leading to inflated performance metrics. There is a critical lack of systematic evaluation for ML pipeline generation beyond English, particularly regarding how resource availability (high vs. low-resource languages) correlates with model performance in complex, multi-step engineering tasks.

Methodology

The authors introduce ML2B, a benchmark designed to evaluate LLMs on generating complete ML pipelines from multilingual natural language descriptions. The methodology encompasses four key pillars:

Dataset Construction:
- Scope: The benchmark comprises 35 Kaggle competitions spanning tabular, text, and image domains.
- Languages: Task descriptions, domains, and data cards were translated into 14 languages (Arabic, Belarusian, Chinese, French, Italian, Japanese, Kazakh, Polish, Romanian, Russian, Spanish, Turkish, Ukrainian), yielding 490 task-language pairs.
- Translation Quality: Translations were performed using GPT-4o and validated by native-speaking researchers with ML expertise. Quality was verified via back-translation BLEU scores (mean >0.35) and inter-annotator agreement (Gwet's AC1 >0.80 for meaning preservation).
- Private Competitions: To mitigate data leakage, 10 of the 35 competitions are "private," meaning they have no publicly available solution code, forum discussions, or significant web indexability, reducing the likelihood of inclusion in LLM training data.
Evaluation Infrastructure:
- Isolated Execution: Evaluation occurs in network-isolated Docker containers. The agent runtime has internet access for code generation, but the grading container is restricted via a forward proxy to essential ML resources (HuggingFace, PyTorch, S3), blocking access to Kaggle and GitHub to prevent runtime solution retrieval.
- Submission Formats: Unlike standard Kaggle submissions (single file), ML2B uses two formats:
  - MONO_PREDICT: A single function returning predictions.
  - MODULAR_PREDICT: A modular format separating train, prepare_val, and predict functions. This structure is designed to reduce preprocessing leakage (e.g., normalizing test data using statistics from the test set).
- Static Leakage Detection: A static analysis tool is employed to detect potential data leakage in the submitted code, flagging instances where training and prediction data are processed together.
Metrics and Baselines:
- Baselines: Two baselines are established for each competition: an LLM-assisted manual baseline (using Gemini 3 Pro for iterative development) and an algorithmic baseline using AutoGluon.
- Performance Measurement: The authors use a performance ratio ( $r_p$ ) comparing model error rates to the baseline. They employ the Area Under the Profile (AUP) metric, specifically a weighted variant ( $AUP_{-1/\ln \tau}$ ), to emphasize the region where models meet or slightly exceed the baseline, distinguishing consistent performers from those with occasional successes.
Experimental Setup:
- Models: Experiments were conducted with GPT-4.1-mini, GPT-OSS-120b, and Gemini-2.5-Flash.
- Frameworks: Models were tested using the AIDE framework and a modified ReAct framework (for GPT-OSS-120b).
- Hardware: Runs were executed on consumer-grade hardware (64GB RAM, single RTX 2080 Ti GPU) to reflect typical research environments.

Key Contributions

First Multilingual ML Pipeline Benchmark: ML2B is the first benchmark to evaluate end-to-end ML pipeline generation across 14 languages, moving beyond the English-only limitation of prior work.
Leakage-Resistant Design: The inclusion of 10 private competitions and the use of network-isolated grading significantly reduce the risk of training data contamination and runtime solution retrieval.
Modular Evaluation Protocol: The introduction of the MODULAR_PREDICT format and static leakage analysis provides a more rigorous method for evaluating code quality and preventing preprocessing leakage compared to single-file submission formats.
Comprehensive Failure Analysis: The study provides a detailed taxonomy of failure modes, including execution errors, constraint violations, and preprocessing inconsistencies.

Results

Experiments with frontier models reveal several critical findings:

Task-Dependent Degradation: Cross-lingual performance degradation is highly task-dependent rather than strictly following traditional resource-availability hierarchies. While some languages show gaps, others perform comparably to or better than English on specific tasks. Variability within resource groups often exceeds variability between them.
Predictors of Variance: Task characteristics, specifically the number of features ( $\rho=+0.50$ ) and domain-specific terminology ( $\rho=+0.44$ ), are stronger predictors of cross-lingual variance than language identity or resource level.
Quality vs. Robustness Trade-off: No single model dominates in both quality and reliability. GPT-4.1-mini achieved the highest average AUP (0.768) but the lowest pass rate (61%), indicating strong ranking ability but difficulty with exact solution generation. Conversely, GPT-OSS-120b (AIDE) showed a high pass rate (81%) but lower AUP.
Primary Bottleneck: The primary failure mode is identified as pipeline instruction-following (e.g., adhering to function signatures, avoiding global variables) rather than language comprehension. Models often fail to follow structural constraints even when they understand the task.
Structured Prompts: The use of structured prompts in ML2B resulted in better performance and stability compared to obfuscated formats used in other benchmarks (e.g., MLE-Bench), confirming that clear task descriptions aid comprehension without enabling memorization shortcuts.

Significance and Claims

The paper claims that ML2B challenges conventional assumptions about multilingual model capabilities. The findings suggest that the assumption that resource availability alone determines multilingual model performance is insufficient; instead, the specific characteristics of the ML task (complexity, terminology, data patterns) play a dominant role.

The authors emphasize the necessity of systematic cross-lingual evaluation for ML pipeline generation to ensure that AI tools are robust and reliable in global contexts. By providing open-source benchmarks, baselines, and infrastructure, ML2B aims to facilitate reproducible research into the cross-lingual robustness of LLMs in engineering-heavy domains. The paper concludes that while current models show promise, the gap between understanding a task and correctly executing a complex, constrained ML pipeline remains a significant hurdle, particularly in non-English contexts.