SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving

SwingArena is a competitive evaluation framework that simulates real-world software development workflows by pairing LLMs as patch submitters and reviewers within a CI-driven pipeline, utilizing a retrieval-augmented code generation module to effectively solve long-context GitHub issues across multiple programming languages.

The Gist

Imagine you are trying to teach a robot how to fix a broken car.

In the past, researchers tested these robots by giving them a single, isolated task: "Here is a loose bolt; tighten it." If the robot tightened the bolt, it got a gold star. This is like the old way of testing AI coding models (called benchmarks like HumanEval). It's simple, but it doesn't tell you if the robot can actually fix a whole engine, or if it will accidentally break the brakes while tightening the bolt.

SWINGARENA is a new, much more realistic "driving test" for AI coding models. Instead of a quiet garage with a single bolt, it simulates a chaotic, high-pressure auto shop where two robots are working together (and against each other).

Here is how it works, broken down into simple concepts:

1. The Two Robots: The "Submitter" and the "Reviewer"

In the real world, when a programmer fixes a bug, they don't just fix it and walk away. They have to convince a human manager (or a team of peers) that the fix is good.

SWINGARENA sets up a battle between two AI models:

• The Submitter (The Mechanic): This robot tries to fix the broken code (the "bug"). It writes a patch (a repair plan).
• The Reviewer (The Inspector): This robot's job is to be skeptical. It tries to break the Submitter's fix. It writes new tests specifically designed to find holes in the repair.

They take turns. The Submitter fixes a problem, the Reviewer tries to break it, the Submitter fixes it again, and so on. It's like a game of "Jailbreak" but for code: one tries to build a wall, the other tries to find a crack in it.

2. The "Giant Library" Problem (Long Context)

Real software isn't just one file; it's a massive library with thousands of books (files). If a robot needs to fix a typo in Chapter 500, it needs to understand how that typo affects Chapter 1, 200, and 800.

Most AI models have a "short attention span" (a limited context window). They can't read the whole library at once.

• The Solution: SWINGARENA uses a special tool called RACG (Retrieval-Augmented Code Generation). Think of this as a super-smart librarian. When the robot asks, "How do I fix this?", the librarian doesn't just dump the whole library on the desk. Instead, the librarian quickly finds the exact three pages the robot needs to read to solve the problem and hands them over. This allows the AI to work on huge, complex projects without getting overwhelmed.

3. The "Real-World" Test (CI Pipeline)

In old tests, if the code ran once without crashing, it was considered a "pass."
In SWINGARENA, the code has to pass a Continuous Integration (CI) pipeline.

• The Analogy: Imagine the robot submits a repair. Before the car can be driven, it must pass a series of automated safety checks: Does the engine start? Do the lights work? Is the paint color correct? Did we accidentally use the wrong type of oil?
• If the robot's fix fails any of these automated checks (even if the code "works" technically), it fails the test. This mimics how real software companies operate, where a fix that breaks the build system is useless.

4. The Results: Who Wins?

The researchers tested top AI models (like GPT-4o, Claude, Gemini, and DeepSeek) in this arena. They found some interesting things:

• The Aggressive Fixers: Some models (like GPT-4o) are great at coming up with bold, creative fixes quickly. They are like mechanics who slap a patch on a hole and hope it holds.
• The Careful Fixers: Other models (like DeepSeek) are slower but more careful. They prioritize making sure the fix doesn't break anything else.
• The Reviewer Effect: The study showed that a model's success depends heavily on who is reviewing it. A "tough" reviewer can expose flaws that a "lenient" one would miss.

Why Does This Matter?

Previous tests were like asking a student to solve a math problem on a piece of paper. SWINGARENA is like putting that student in a real classroom, giving them a group project, a strict teacher, and a ticking clock.

It reveals that while AI is getting better at writing code, it still struggles with the messy, collaborative, and safety-critical reality of professional software engineering. SWINGARENA helps us see exactly where the robots are failing so we can teach them better.

In short: SWINGARENA stops treating AI like a calculator and starts treating it like a junior engineer, testing if it can actually survive the chaos of a real software team.

Technical Summary

Here is a detailed technical summary of the paper "SWINGARENA: ADVERSARIAL PROGRAMMING ARENA FOR LONG-CONTEXT GITHUB ISSUE SOLVING".

1. Problem Statement

Current benchmarks for Large Language Models (LLMs) in software engineering (e.g., HumanEval, MBPP, SWE-Bench) suffer from three critical limitations that fail to reflect real-world development:

• Static and Predictable: They rely on fixed, one-shot coding tasks that do not capture the dynamic, iterative nature of software development where code is constantly reviewed and refined.
• Lack of Adversarial Interaction: They evaluate single agents in isolation, ignoring the critical "submitter-reviewer" loop where a developer proposes a fix and a reviewer (or CI system) challenges it with new tests.
• Insufficient Context and CI Integration: They often ignore the complexity of large codebases (long-context challenges) and omit the full Continuous Integration (CI) pipeline, focusing merely on unit test pass rates rather than build stability, style compliance, and security gates.

The paper argues that a robust evaluation must simulate the collaborative, iterative, and adversarial workflow of professional software engineering, including long-context reasoning and automated CI validation.

2. Methodology: SWINGARENA Framework

SWINGARENA is an adversarial evaluation framework that operationalizes full CI workflows as dynamic testing arenas. It introduces a dual-agent system and a retrieval-augmented generation pipeline.

A. Data Construction

The authors curated a high-quality dataset from 2,300 real-world GitHub issues across four languages: C++, Python, Rust, and Go.

• Filtering Pipeline:
  1. Repository Mining: Selected high-popularity repositories to ensure code quality.
  2. CI Test Filtering: Retained only issues where the original Pull Request (PR) passed all CI checks (GitHub Actions, Travis CI), ensuring the "golden" solution is valid.
  3. LLM-as-a-Judge: Used an LLM (Grok-3-beta) to assess problem clarity and difficulty.
  4. Expert Filtering: Human experts validated the LLM's assessments to remove hallucinations and low-quality instances.
• Final Dataset: 400 evaluation instances (100 per language) and a 100-sample ablation split.

B. The Adversarial Arena (Battle Protocol)

The core of SWINGARENA is a multi-round "battle" between two agents:

1. Submitter: Generates a code patch to fix a reported issue.
2. Reviewer: Generates unit tests specifically designed to expose flaws, edge cases, or regressions in the submitter's patch.

• Interaction: Agents alternate roles over multiple rounds (e.g., 10 rounds total).
• Scoring:
  • Submitter (+1): If the patch passes all CI checks, including the reviewer's new tests.
  • Reviewer (+1): If their test successfully fails the submitter's patch (revealing a bug).
  • Penalties: -1 for failures in compilation, style, or logic.
• Verification: All evaluations run inside isolated Docker containers executing the repository's native CI pipelines (e.g., cargo for Rust, go test for Go) to ensure reproducibility and strict adherence to project-specific constraints.

C. Retrieval-Augmented Code Generation (RACG)

To handle the "long-context" challenge of large codebases, SWINGARENA employs a specialized RACG module:

• File Retriever: Uses BM25 (sparse retrieval) to rank source files based on lexical similarity to the problem description, pruning the search space.
• CodeChunker: Decomposes files into semantically meaningful units (functions, classes, blocks) using syntax-aware parsing (AST) or regex heuristics.
• CodeReranker: Uses CodeBERT to perform dense reranking of chunks based on cosine similarity, incorporating tie-breaking rules (favoring definitions over usages) and proximity bias.
• Token Budget-Aware Packing: Dynamically selects and packs chunks to fit within the model's context window, prioritizing the most relevant code while respecting token limits.

3. Key Contributions

1. Adversarial CI Evaluation Protocol: A novel framework that pairs submitters and reviewers in a dynamic loop, simulating real-world PR workflows with role-switching and strict CI-based scoring.
2. Multi-Language Long-Context Pipeline (RACG): A robust, language-agnostic retrieval system (supporting C++, Python, Rust, Go) that combines sparse and dense retrieval with syntax-aware chunking, serving as a strong baseline for long-context code tasks.
3. Curated Dataset: A high-quality, CI-grounded dataset of 2,300 GitHub issues with verified solutions, including scripts for reproducible retrieval and CI execution.
4. Comprehensive Evaluation: Evaluation of state-of-the-art models (GPT-4o, Claude-3.5, Gemini-2.0, DeepSeek-V3) and open-source models, revealing distinct behavioral patterns.

4. Experimental Results

The authors evaluated multiple proprietary and open-source models using the SWINGARENA framework.

• Model Performance Trends:
  • GPT-4o: Demonstrated "aggressive patching," achieving high win rates (≥0.90) as a submitter against various reviewers, though with slightly lower CI pass rates (SPR/RPR) compared to others, suggesting a trade-off between assertiveness and strict correctness.
  • DeepSeek & Gemini: Prioritized correctness and CI stability, achieving the highest CI pass rates (up to 0.66) but slightly lower win rates in adversarial matchups.
  • Self-Consistency: All models showed high win rates when reviewing their own submissions (e.g., Claude 1.00, GPT-4o 0.97), indicating strong internal alignment between generation and testing capabilities.
• Language-Specific Insights:
  • Models generally performed best on C++ and relatively worse on Rust and Python.
  • DeepSeek showed the most balanced multi-language reasoning ability.
• Ablation Studies (RACG):
  • Incorporating RACG significantly improved performance. For example, in C++, RACG increased Best@3 from 0.38 to 0.42 and Win Rate from 0.77 to 0.84.
  • RACG outperformed simple BM25 retrieval and Top-k baselines, proving that syntax-aware chunking and dense reranking are crucial for long-context tasks.
• Failure Analysis:
  • Cross-File Consistency (31%): Models struggled with fixes spanning multiple files.
  • Context Retrieval (26%): Failures often stemmed from retrieving irrelevant code or missing critical context in large repos.
  • Non-Functional Violations (24%): Code often failed style, security, or linting gates despite being functionally correct.

5. Significance and Impact

• Realism: SWINGARENA bridges the gap between static benchmarks and real-world software engineering by enforcing full CI pipelines, style guides, and adversarial testing.
• Diagnostic Power: The framework reveals nuanced trade-offs in model behavior (e.g., aggressiveness vs. stability) that static benchmarks miss, providing actionable insights for model developers.
• Scalability: The modular design allows for easy extension to new languages and repositories, offering a scalable approach for evaluating LLMs in CI-driven environments.
• Future Directions: The paper highlights the need for better architectural reasoning (multi-file consistency) and improved retrieval strategies for extremely large, poorly structured codebases.

In conclusion, SWINGARENA represents a significant step forward in evaluating LLMs for software engineering, moving beyond "does the code run?" to "does the code survive a rigorous, collaborative, and adversarial professional workflow?"