💻 computer science

SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agents

The paper introduces SWE-Future, a method that synthesizes realistic, future-oriented software engineering tasks by using pre-existing repository data to forecast upcoming feature, bugfix, and refactor needs, thereby creating a benchmark that avoids the data contamination risks of replaying historical pull requests while maintaining high relevance to actual repository evolution.

Original authors: Qiao Zhao, JianYing Qu, Jun Zhang, Yehua Yang, Hanwen Du, Zhongkai Sun

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qiao Zhao, JianYing Qu, Jun Zhang, Yehua Yang, Hanwen Du, Zhongkai Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to fix a car.

The Old Way (The "Replay" Problem)
Most current methods teach the robot by showing it a video of a mechanic fixing a car yesterday. The robot watches the video, memorizes the steps, and then takes a test where it has to fix the exact same car in the exact same way.

The Flaw: If the robot has already seen that video while it was learning its basic language (pre-training), it doesn't actually know how to fix cars; it just remembers the answer. It's like cheating on a math test because you memorized the answer key.
The Alternative: Some people try to invent fake car problems from scratch. But these fake problems often ignore reality. They might ask the robot to fix a part that doesn't exist in that specific car model, or they might forget that the mechanic needs a specific wrench. The robot gets confused because the problem feels "plausible" but isn't "real."

The New Way: SWE-Future (The "Weather Forecast" Method)
The authors of this paper propose a middle ground called SWE-Future. Instead of showing the robot a video of a past repair, they teach it to predict what repairs will be needed next, and then create a test based on that prediction.

Here is how it works, step-by-step:

1. The "Weather Forecast" (Prediction)

Imagine you are looking at a car that has been driven for a while. You see the dashboard lights flickering, the tires are wearing unevenly, and the owner has been complaining about a weird noise.

The Method: The system looks only at the evidence available before a specific date (like looking at the car's history up to yesterday).
The Prediction: Based on that evidence, it makes a "forecast" of what will likely go wrong next. It doesn't guess the exact fix; it guesses the category of the problem.
- Example Forecast: "This car is likely to need a fix for its 'brake sensor' soon," or "It will need a new 'radio feature'."
The Key: It creates a list of "likely future tasks" without ever looking at the actual repair that happened later.

2. The "Reality Check" (Validation)

After making these forecasts, the researchers wait six months. They look at the car's actual repair logs from that period.

The Check: Did the car actually have brake sensor issues? Did they add a radio feature?
The Result: They found that their "weather forecast" was right about 58% of the time. This proves that looking at past patterns can successfully predict future needs.

3. Building the Test (Synthesis)

This is the magic part. Now, the researchers want to create a test for the robot.

The Trap: They do not use the actual repair log from the last six months. That would be cheating (replaying history).
The Solution: They take the forecast (e.g., "Fix the brake sensor") and apply it to the car's current state (as it exists today).
The Result: They build a brand new, unique repair task that the robot has never seen before. The robot has to figure out how to fix the brake sensor based on the car's current blueprints, not by remembering a video of someone else doing it.

Why This Matters

Think of it like a cooking competition.

Old Way: The chef is asked to recreate a dish they saw on TV last week. If they saw the show before, they win by memory, not skill.
SWE-Future Way: The judges look at the ingredients the chef has in their pantry and predict, "You are likely to run out of salt soon." The judges then ask the chef to create a new recipe that solves a salt shortage. The chef has to use their actual cooking skills to invent a solution, because they can't just copy a recipe from a TV show they might have seen.

The Bottom Line

The paper claims that by using forecasts to guide the creation of tasks, they can build a dataset of 200 realistic coding challenges across 61 different software projects.

These tasks are realistic because they are based on actual project trends.
They are fair because the robot cannot cheat by memorizing the answer key (the historical repair log), since the answer key was never shown during the task creation.
They are future-oriented, preparing the robot for problems that haven't happened yet, rather than just repeating the past.

In short: SWE-Future teaches robots to anticipate problems and solve them fresh, rather than just memorizing how problems were solved in the past.

Technical Summary: SWE-Future

Problem Statement

Current benchmarks for software engineering (SWE) agents often rely on replaying historical GitHub issues and pull requests (PRs). While this provides realism, it introduces a critical validity problem: as benchmarks become public, their repositories, issue texts, and PR metadata risk appearing in model pretraining, fine-tuning, synthetic-data generation, or model-selection loops. This creates data contamination, where models may simply memorize benchmark solutions rather than demonstrating genuine coding capabilities. Conversely, purely synthetic task generation avoids historical replay but often drifts from real repository needs, ignoring project conventions, maintainer priorities, and realistic dependency constraints.

Methodology

SWE-Future proposes a "forecast-conditioned data synthesis" method to bridge this gap. Instead of replaying realized PRs, the method predicts likely future repository evolution based on pre-snapshot evidence and uses these predictions to guide task synthesis. The process is divided into four distinct stages with strict temporal boundaries:

Evidence Collection (Pre- $T_0$ ): For a repository snapshot at time $T_0$ , the system aggregates pre- $T_0$ issues, PRs, labels, and text. It filters out maintenance-only changes (e.g., dependency bumps, documentation, CI) to focus on core coding tasks: feature implementation/enhancement, bugfix, and refactor.
Forecasting: The system clusters pre- $T_0$ signals into "task families" (e.g., "recurring hangs in module X" or "API extension for feature Y"). These are not specific PR titles but coarse directions of repository demand. A deterministic scoring heuristic ranks these clusters based on signal volume, issue count, and label diversity.
Retrospective Validation: After forecasts are frozen, the system looks at PRs merged in the window $(T_0, T_1]$ (where $T_1 = T_0 + 6$ months). It uses semantic matching to determine if the frozen forecast families align with actual future work. Crucially, these later PRs are only used to validate the forecast direction; they are not used to construct the final tasks.
Task Synthesis ( $T_{gen}$ ): Validated forecast families serve as conditioning signals. A task-generation snapshot ( $T_{gen}$ $T_{g e n}$ ) is selected. Using only the repository state visible at $T_{gen}$ $T_{g e n}$ and the validated family direction, a multi-agent workflow constructs specific coding tasks. This workflow includes:
- Context Selection: Identifying concrete code targets within $T_{gen}$ .
- Task Writing: Generating an issue-style request visible to the agent.
- Oracle Design & Patch Construction: Creating a hidden test patch and a gold patch (solution) from scratch, ensuring the task is executable (FAIL TO PASS).
- Verification: Validating that the test patch fails on the base snapshot and passes with the gold patch, while ensuring no leakage of hidden artifacts into the public task description.

Key Contributions

Forecast-Conditioned Synthesis: Introduces a method for constructing realistic, future-oriented coding tasks by using repository-evolution forecasts as conditioning signals, rather than relying on historical PR replay.
Retrospective Validation Procedure: Establishes a protocol to test whether forecast families capture future repository work. In an 80-repository study, the forecaster achieved 58.1% future-work relevance (strong or related matches) under a semantic matching metric.
Executable Dataset Construction: Demonstrates the conversion of validated forecasts into a 200-task coding-agent dataset across 61 repositories. These tasks are grounded in $T_{gen}$ snapshots and include executable validation assets (test and gold patches) constructed without using post- $T_0$ PR artifacts.

Results

Forecasting Performance: Out of 260 forecast families generated from 76 repositories, 151 (58.1%) were validated as "strong" or "related" to future work. Bugfix families showed the highest relevance (89/139), followed by feature implementation/enhancement (45/93).
Dataset Composition: The final dataset contains 200 tasks: 120 bugfix, 60 feature implementation/enhancement, and 20 refactor tasks.
Contamination Mitigation: The method successfully decouples task generation from historical PR replay. The public task descriptions do not contain test-patch logic, gold-patch details, or retrospective validation labels, which remain hidden evaluation assets.

Significance and Claims

The paper claims that SWE-Future addresses the "middle ground" between historical replay (high realism, high contamination risk) and pure synthetic generation (low contamination, low repository pressure). By freezing repository history at $T_0$ , forecasting likely directions, and grounding tasks in a later snapshot $T_{gen}$ , the method reduces direct dependence on historical pull-request replay.

The authors emphasize that the goal is not to prove models have never seen a repository, but to avoid the direct replay of observed historical PRs as task material. The approach allows for the creation of repository-specific tasks that reflect natural project evolution while maintaining a clear information boundary to prevent data leakage. The paper acknowledges limitations, noting that forecasters may pick up noisy anchors and that generated test patches may lack the precision of maintainer-written tests, particularly for refactor tasks. Future work is suggested to improve forecast noise filtering and test synthesis quality.