Technical Summary: Multi-LCB

Problem Statement

While Large Language Models (LLMs) have demonstrated significant capabilities in code generation, existing benchmarks suffer from critical limitations. Standard datasets like HumanEval, MBPP, and APPS are often static, prone to training data contamination, and heavily biased toward Python. LiveCodeBench (LCB) addressed contamination and scope issues by curating competitive programming problems with release-date filtering; however, it remains restricted to Python. This limitation obscures a fundamental reality of software engineering: developers work across diverse languages with distinct syntax, semantics, and idioms. Consequently, it remains unclear whether LLMs possess genuine cross-lingual coding competence or if their performance is merely a result of Python overfitting and language-specific data leakage.

Methodology

The authors introduce Multi-LCB, a benchmark that extends the LiveCodeBench framework to twelve programming languages: C++, C#, Python, Java, Rust, Go, TypeScript, JavaScript, Ruby, PHP, Kotlin, and Scala. The core methodology involves transforming existing Python-centric LCB tasks into equivalent tasks in other languages while strictly preserving LCB's contamination controls and evaluation protocols.

Key technical components include:

• Task Conversion Pipeline: The benchmark converts LCB's "Functional" format (specific to LeetCode, where a function is implemented) into a unified STDIN/STDOUT format. This is achieved via an automatic pipeline that:
  1. Adapts natural language problem statements and examples for model prompts.
  2. Converts all test cases (public and hidden) from functional assertions to language-agnostic input/output streams.
  3. Ensures tasks remain language-agnostic, as the source problems from LeetCode, AtCoder, and Codeforces are authored to avoid language-specific ambiguities.
• Evaluation Protocol: The system adopts a zero-shot prompting strategy. Models are instructed to act as experts in the target language and generate a complete program reading from standard input and writing to standard output.
• Execution Environment: Code is executed in isolated sandbox containers with strict resource limits (6s wall-time, 4GB memory) using specific compilers/interpreters (e.g., GCC 13 for C++, OpenJDK 21 for Java).
• Metrics: Performance is measured using Pass@1 (the fraction of tasks solved correctly by the first generated solution) across hidden test suites, averaged over 10 runs.

The study evaluated 24 diverse LLMs (ranging from 7B to 685B parameters), including instruction-tuned and reasoning-augmented variants from families such as Qwen3, DeepSeek, and GPT-OSS.

Key Results

The evaluation of 24 models on the Multi-LCB dataset (covering tasks released between February and May 2025) yielded several critical findings:

1. Python is Not a Reliable Proxy: Strong performance in Python does not guarantee competence in other languages. Models that outperform others in Python often lose their advantage in languages like Go, Rust, or Scala. For instance, GPT-OSS-120B (Medium) outperformed Qwen3-235B in several non-Python languages despite the latter's superior Python scores.
2. Python Overfitting: There is a consistent bias toward Python. Scatter plots of Python Pass@1 versus average cross-language Pass@1 show that almost all models lie above the diagonal, with some reasoning-augmented models showing gaps exceeding 60% in Python versus under 30% in other languages.
3. Language-Specific Contamination: Evidence of data leakage varies significantly by language, reflecting uneven distributions in pretraining corpora. Older tasks (pre-cutoff) showed inflated scores, while post-cutoff tasks revealed true generalization capabilities.
4. Performance Disparities: A clear difficulty gradient exists across languages. Python achieved the highest mean Pass@1 (48%), followed by Java and C++ (44%). Statically typed or less prevalent languages like Scala consistently trailed with means below 29%.
5. Benchmark Fidelity: Comparisons with the original LiveCodeBench leaderboard showed high fidelity, with differences typically within a few percentage points, confirming that the conversion process did not introduce artificial difficulty for Python tasks.

Significance and Claims

The paper claims that Multi-LCB establishes a rigorous, contamination-aware benchmark for evaluating multilingual code generation. Its primary significance lies in:

• Addressing the Python Limitation: It directly challenges the assumption that Python performance is sufficient to gauge general coding ability, exposing critical gaps in current LLM capabilities regarding cross-language generalization.
• Systematic Assessment: By maintaining compatibility with the original LCB format and update cycle, Multi-LCB enables the systematic tracking of future model improvements across multiple languages.
• Reproducibility: The authors publicly release all prompts, source code, and experimental configurations to facilitate future research.

The authors conclude that current LLMs exhibit substantial performance disparities across languages and that achieving robust, language-agnostic coding competence remains a significant challenge, necessitating increased training coverage of non-Python languages.