🤖 machine learning

Merge-Bench: Resolve Merge Conflicts with Large Language Models

This paper introduces Merge-Bench, a large-scale dataset of real-world merge conflicts, and presents LLMergeJ, a Java-specific model trained with Group Relative Policy Optimization that outperforms several commercial LLMs, though even the best models currently resolve fewer than 60% of conflicts across 11 programming languages.

Original authors: Benedikt Schesch, Michael D. Ernst

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Benedikt Schesch, Michael D. Ernst

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are working on a group project where two friends are editing the same document at the same time. Friend A changes a paragraph, and Friend B changes the exact same paragraph. When you try to combine their work, the computer gets confused and screams, "I don't know which version to keep!" This is called a merge conflict.

Usually, a human has to step in, read both versions, figure out what the friends actually meant to do, and manually fix the mess. This takes time and can lead to mistakes.

This paper introduces a new way to teach computers to fix these messes automatically using "smart" AI (Large Language Models). Here is the breakdown of their work in simple terms:

1. The Problem: The Computer is Clueless

Traditional tools are like a robot that only looks at the letters. If Friend A wrote "cat" and Friend B wrote "dog," the robot just sees a clash. It doesn't understand that maybe they were talking about pets, or maybe one was a typo. It needs a human to explain the intent.

2. The Solution: A New Training Ground (Merge-Bench)

To teach an AI to fix these conflicts, you need a massive library of examples showing "Here was the mess, and here is how a human expert fixed it."

The Old Way: Other researchers tried to make these libraries by hand or by running tests. This was slow, expensive, and sometimes the AI would "cheat" by memorizing the test answers rather than learning to fix the code.
The New Way (Merge-Bench): The authors built a giant, automated library called Merge-Bench. They scraped 7,938 real-life conflicts from 1,439 different public code projects on GitHub.
- The Analogy: Imagine a teacher who doesn't need to grade every single homework assignment by hand. Instead, they have a machine that automatically grabs 8,000 real-world examples of students making mistakes and the correct answers written by the teacher. Because the machine does all the work, they can have a library that is huge and never runs out.

3. The Student: LLMergeJ

The authors trained a specific AI model named LLMergeJ (the "J" stands for Java, the programming language they focused on).

How they taught it: Instead of just showing the model the answer (like a standard teacher), they used a method called Reinforcement Learning.
- The Analogy: Think of it like training a dog. If the dog guesses the right trick, it gets a treat. If it guesses wrong, it gets nothing. If it refuses to guess and just says "I don't know" (preserving the conflict), it gets a tiny crumb.
- The AI tried thousands of times. When it figured out the right way to combine the code, it got a big reward. Over time, it learned not just what the answer looked like, but how to think about the problem to get there.

4. The Results: Small Dog, Big Tricks

They tested their 14-billion-parameter model (which is relatively small for AI standards) against the biggest, most expensive commercial AI models available (like Gemini, Claude, and Grok).

The Surprise: Their small, custom-trained model did better than almost all the giant commercial models. It solved about 59% of the conflicts correctly (after ignoring minor formatting differences).
The Comparison: The best commercial model (Gemini 2.5 Pro) was slightly better, but the authors' model beat the others by a significant margin.
The Lesson: A small model trained specifically on how to fix merge conflicts using rewards is better than a giant, general-purpose model that was just asked to do it once.

5. Why This Matters

No Cheating: Their testing method doesn't rely on running code tests (which AI can sometimes trick). It compares the code directly to what the human developer actually did.
Scalable: Because they didn't need humans to label the data, they could make the training set as big as they wanted.
Language Agnostic: While they only trained the model on Java, they tested the big commercial models on 11 different languages (C, Python, Rust, etc.). They found that the AI's behavior was similar across all languages, suggesting their method could work for any programming language.

Summary

The authors built a massive, automated gym (Merge-Bench) to train a specific AI athlete (LLMergeJ) to solve code conflicts. By using a reward-based training system, they taught a relatively small AI to outperform the biggest, most expensive AI models at this specific task, proving that specialized training beats general size when it comes to fixing software messes.

`), 0 otherwise.
* $R_{format}$ : 1 if the output uses correct Markdown formatting, 0 otherwise.
* $R_{resolution}$ : 1.0 for exact textual match, 0.5 for normalized source code match, 0.1 for preserving the conflict (no resolution), and 0 for incorrect merges.

Advantage over SFT: Unlike Supervised Fine-Tuning (SFT), which can only learn from final outputs, GRPO optimizes the reasoning process and handles the unpredictability of ground truth (where multiple syntactically different solutions may be valid) by using group-relative advantages. This prevents the model from overfitting to specific syntactic patterns.

Key Contributions

Merge-Bench: A large-scale, scalable benchmark resistant to reward hacking and requiring zero manual annotation.
Test-Free Evaluation: A methodology that bypasses test execution, enabling scalable training and evaluation without the vulnerabilities of test-suite-based rewards.
Human-Filter-Free SWE-RL: A reinforcement learning task for software engineering that utilizes internet-scale data without human filtering or manual adjustments.
LLMergeJ: The first model trained with online RL for merge conflict resolution, demonstrating that a 14B parameter model can outperform significantly larger commercial models.

Results

Java Performance

On the Java subset of Merge-Bench (806 conflicts):

LLMergeJ (14B) achieved 48.8% exact textual match and 58.9% normalized source code match.
Comparison: LLMergeJ outperformed three major commercial LLMs (Claude Opus 4, o3 Pro, and Grok 4) and a Qwen3-14B SFT baseline. It trailed only Gemini 2.5 Pro (62.5% normalized match).
SFT Baseline: Even when the SFT baseline was trained on the same data with hyperparameters optimized specifically for the test set, it achieved only 36.6% exact match and 44.7% normalized match, highlighting the superiority of the GRPO approach for this task.
Behavioral Analysis:
- Aggressive Resolvers (LLMergeJ, Gemini 2.5 Pro, o3 Pro): High confidence, resolving 51–62% of conflicts, with low conflict preservation rates (3–21%).
- Conservative Resolvers (Grok 4, Llama 4 Maverick): High conflict preservation rates (40–86%), indicating uncertainty.
- Format-Struggling Models: Some models (e.g., QwQ 32B, R1 1.5B) failed to follow Markdown formatting instructions, leading to invalid outputs.

Cross-Language Performance

The authors evaluated six state-of-the-art commercial models across 11 languages (C, C++, C#, Go, Java, JavaScript, PHP, Python, Ruby, Rust, TypeScript).

Language Difficulty: C was found to be the most difficult to merge, while Java, PHP, and Python were the easiest.
Stability: Commercial LLM performance was largely stable across languages.
Generalization: The behavioral patterns (e.g., which models are aggressive vs. conservative) remained consistent across languages, suggesting the techniques may generalize, though LLMergeJ itself was only trained on Java.

Significance and Claims

The paper claims that LLMergeJ demonstrates that small, specialized models trained with reinforcement learning can achieve competitive or superior performance compared to massive commercial models for specific software engineering tasks. The work highlights that:

Reinforcement Learning (GRPO) is superior to SFT for merge conflicts because it optimizes reasoning and handles the ambiguity of multiple valid solutions without degrading performance.
Scalability is achievable without human annotation, addressing a bottleneck in current SWE benchmarks like SWE-Bench.
Parameter Efficiency: A 14B model can rival models with orders of magnitude more parameters when trained with the correct methodology and data.

The authors maintain a modest stance, acknowledging limitations such as the potential for ground truth errors, the possibility of data leakage in commercial training sets, and the fact that the training was limited to Java. They note that the size constraints of the dataset may exclude the most complex merge problems, potentially leading to an overestimation of performance on all real-world conflicts.