🤖 AI

MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering

This paper introduces MEnvAgent, a scalable multi-language framework that automates the construction of verifiable software engineering environments through a multi-agent architecture and environment reuse mechanism, thereby overcoming dataset scarcity and enabling the creation of the largest open-source polyglot dataset for LLM agents.

Original authors: Chuanzhe Guo, Jingjing Wu, Sijun He, Yang Chen, Zhaoqi Kuang, Shilong Fan, Bingjin Chen, Siqi Bao, Jing Liu, Hua Wu, Qingfu Zhu, Wanxiang Che, Haifeng Wang

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chuanzhe Guo, Jingjing Wu, Sijun He, Yang Chen, Zhaoqi Kuang, Shilong Fan, Bingjin Chen, Siqi Bao, Jing Liu, Hua Wu, Qingfu Zhu, Wanxiang Che, Haifeng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef (an AI) trying to recreate a famous dish from a recipe book. The recipe says, "Add salt, pepper, and a specific type of rare herb." But here's the catch: before you can even start cooking, you have to build a kitchen from scratch. You have to find the right stove, install the correct gas pipes, buy the exact brand of salt, and ensure the rare herb grows in your garden.

If you get the stove wrong, the food burns. If you buy the wrong salt, the dish tastes terrible. In the world of software engineering, this "kitchen building" is called Environment Construction.

For a long time, teaching AI to fix software bugs was like asking a chef to cook without a kitchen. We had the recipes (the code), but we couldn't easily build the kitchens (the software environments) to test if the AI's fixes actually worked. Building these kitchens was slow, expensive, and only worked for a few types of "cuisines" (programming languages).

Enter MEnvAgent: The Ultimate Kitchen Builder

The paper introduces MEnvAgent, a new system designed to automatically build these software kitchens for 10 different programming languages (like Python, Java, C++, etc.). It's like having a team of specialized construction workers who can build a kitchen for any recipe, no matter how complex.

Here is how MEnvAgent works, using simple analogies:

1. The Three-Step Construction Crew

Instead of one person trying to do everything, MEnvAgent uses a team of three "agents" (AI specialists) working in a loop:

The Architect (Planning): This agent reads the recipe and the blueprint. It figures out what stove, pipes, and ingredients are needed.
The Builder (Execution): This agent actually starts building. It installs the software and runs the tests. If a pipe leaks or a part doesn't fit, this agent tries to fix it on the fly.
The Inspector (Verification): Once the kitchen is built, this agent tries to cook the dish. If the dish tastes right (the code passes the test), the job is done. If it fails, the Inspector tells the Architect exactly what went wrong so they can try again.

2. The "Reuse" Trick (Saving Time and Money)

Building a kitchen from scratch every single time is a waste of time. Imagine if you had to build a new house every time you wanted to bake a cake.

MEnvAgent has a clever trick called the Environment Reuse Mechanism.

The Analogy: Imagine you need to build a kitchen for a new pizza recipe. Instead of starting with an empty lot, MEnvAgent looks at its library of previously built kitchens. It finds one that is already 90% built for a similar pizza.
The Patch: It then sends a "patch crew" (the EnvPatchAgent) to make small, quick adjustments to that existing kitchen to fit the new recipe. This is much faster than building from the ground up.
The Result: The paper claims this saves 43% of the time compared to building everything from scratch.

3. The Proof: MEnvBench and MEnvData

To prove their system works, the authors didn't just guess; they built a giant testing ground called MEnvBench.

Think of this as a massive "Cooking Olympics" with 1,000 challenges across 10 different languages.
They tested MEnvAgent against other systems and found it was the winner. It fixed more "broken kitchens" (improving success rates by 8.6%) and did it much faster.

4. The Big Payoff: A Library of Perfect Kitchens

Because MEnvAgent is so good at building these environments, the authors used it to create MEnvData-SWE.

The Analogy: This is like a massive library of 3,000+ perfectly built kitchens, each with a working recipe and a test result.
The Impact: They used this library to train other AI chefs. The result? The AI chefs got significantly better at fixing software bugs. The paper shows that models trained on this data performed much better on real-world software tasks than models trained without it.

Summary

In short, MEnvAgent solves the problem of "how do we test if an AI can fix code?" by automatically building the necessary testing environments for many different programming languages. It does this by using a team of AI agents to plan, build, and check, and by cleverly reusing old environments to save time. The result is a faster, more reliable way to train AI to be better software engineers.

What the paper does NOT claim:

It does not claim this system can fix every bug in the world instantly.
It does not claim this replaces human engineers entirely; it is a tool to help build better testing grounds for AI.
It does not discuss using this for medical software or critical safety systems specifically, other than noting the need for safety sandboxes (isolated kitchens) to prevent malicious code from running.

Technical Summary: MEnvAgent

Problem Statement

The advancement of Large Language Model (LLM) agents for Software Engineering (SWE) is currently bottlenecked by the scarcity of verifiable datasets. While benchmarks like SWE-bench exist, constructing the executable environments required to validate solutions is a significant challenge. Existing approaches face a dilemma:

Static metrics scale efficiently but provide only approximate verification signals.
Manual construction ensures quality but is labor-intensive and largely restricted to Python.
Automated attempts struggle with the complexity of diverse dependencies, version conflicts, and inconsistent testing protocols across non-standard repositories, leading to low success rates and prohibitive time costs due to the need for "clean-slate" rebuilds upon failure.

There is a critical gap in scalable, verifiable support across diverse programming languages (polyglot environments).

Methodology: MEnvAgent

To address these challenges, the authors introduce MEnvAgent, a multi-language framework for automated environment construction. The system is built on two core pillars:

1. Multi-Agent Planning-Execution-Verification Architecture

MEnvAgent employs a closed-loop architecture where specialized agents iteratively diagnose and resolve construction failures:

Planning Stage:
- Repository Analysis Agent: Explores file structures and dependencies to summarize project types and entry points.
- Environment Setup Agent: Selects the optimal base image and drafts the installation script ( $P$ ).
- Test Configuration Agent: Synthesizes a compatible test script ( $T$ ) ensuring verification logic aligns with the environment.
Execution Stage:
- Environment Execution Agent: Instantiates a container and executes the build process. It monitors terminal output in real-time to dynamically adjust commands (e.g., fixing missing packages) to resolve immediate errors. If failures persist, it reverts to the Planning Stage to regenerate a strategy.
Verification Stage:
- Verification Agent: Executes the test script. It performs error attribution to determine if failures stem from missing dependencies or incorrect test commands, feeding diagnostic feedback back to the Planning Stage for iterative refinement.

2. Environment Reuse Mechanism

To mitigate the high computational cost of building environments from scratch, MEnvAgent introduces a novel reuse strategy:

Retrieval: Instead of starting from a raw base image, the system searches an Environment Pool of previously verified environments. It uses a hierarchical retrieval strategy based on Version Consistency (exact version match) and Backward Compatibility (selecting newer environments that support older dependencies).
Adaptation: Once a similar historical environment ( $S_{sim}$ ) is found, an EnvPatchAgent generates an incremental command sequence ( $\Delta P$ ) to adapt the environment to the target repository snapshot. This avoids full rebuilds, significantly reducing overhead.

Key Contributions

The paper presents three primary contributions:

MEnvAgent Framework: A multi-agent system covering 10 programming languages based on a Planning-Execution-Verification loop, featuring a novel environment reuse mechanism to reduce computational overhead.
MEnvBench: The first comprehensive benchmark for evaluating multi-language executable environment construction. It comprises 1,000 tasks across 10 mainstream languages and 200 open-source repositories, featuring strict execution-based evaluation and quality assurance.
MEnvData-SWE: The largest open-source polyglot dataset of realistic, verifiable Docker environments to date. Constructed via MEnvAgent, it includes solution trajectories that enable performance gains for LLMs on downstream SWE tasks.

Experimental Results

The framework was evaluated on MEnvBench using two backbone LLMs: Kimi-K2 (open-source) and Gemini-3-Flash (closed-source).

Performance vs. Baselines: MEnvAgent outperformed state-of-the-art baselines (SWE-Factory, SWE-Bench-Live, Repo2Run) across all languages.
- Fail-to-Pass (F2P) Rate: Improved by 8.6% on average.
- Pass Rate: Improved by 11.0% on average.
- Time Cost: Reduced by 43.0% on average.
Ablation Studies:
- Removing the Environment Reuse mechanism (building from scratch) increased time costs by 46% and dropped the Pass Rate by 18.5%.
- Removing the EnvPatchAgent (direct reuse without patching) caused the Reuse Success Rate to drop from 39% to 25%, leading to frequent fallbacks to scratch builds.
- Scalability: As the number of historical instances per repository increased (from 1 to 10), the Reuse Success Rate rose to 39%, demonstrating that the framework becomes more efficient with larger data scales.
Downstream Utility: Fine-tuning various models (Qwen2.5-Coder, Qwen3-Coder, GLM-4.5-Air) on solution trajectories from MEnvData-SWE resulted in substantial performance gains on SWE-bench Verified and SWE-bench Multilingual benchmarks. Notably, the fine-tuned Qwen2.5-Coder-32B matched GPT-4.1 on SWE-bench Verified.

Significance and Claims

The authors claim that MEnvAgent successfully bridges the gap in scalable, verifiable support for polyglot software engineering. By automating the traditionally labor-intensive task of environment construction, the framework lowers the barrier for researchers to conduct large-scale, multi-language SWE studies.

The paper emphasizes that the Environment Reuse Mechanism is a critical innovation that transforms environment construction from a linear, costly process into an incremental, efficient one. Furthermore, the release of MEnvData-SWE provides the community with a resource to train more robust and versatile coding agents, effectively validating the utility of execution-verified data in pushing the performance boundaries of SOTA models.

The authors acknowledge potential risks regarding automated code execution but note that the framework operates within strictly isolated Docker sandbox environments to ensure safety. All code, benchmarks, and datasets are open-sourced to facilitate future research.