CONCUR: Benchmarking LLMs for Concurrent Code Generation

This paper introduces CONCUR, a novel benchmark comprising 115 concurrency-specific problems designed to evaluate and highlight the limitations of Large Language Models in generating complex concurrent code, addressing a critical gap left by existing benchmarks that focus solely on sequential code.

The Gist

Imagine you have a team of incredibly smart, super-fast robots (Large Language Models, or LLMs) that are learning to write computer code. We've already tested them on writing sequential code—which is like telling a robot to make a sandwich step-by-step: Get bread, put peanut butter, put jelly, close sandwich. If the robot follows the steps, the sandwich is good.

But what happens when you ask these robots to write concurrent code?

The Problem: The Busy Kitchen Analogy

Concurrent code is like a busy restaurant kitchen where five chefs are trying to cook a meal at the same time.

• They all need to use the same stove.
• They all need to grab the same knife.
• They are all shouting orders at once.

If they aren't perfectly coordinated, chaos ensues:

• Deadlock: Chef A is waiting for Chef B to put down the knife, but Chef B is waiting for Chef A to turn off the stove. They both freeze. The meal never gets made.
• Race Condition: Two chefs try to add salt to the soup at the exact same millisecond. One adds a teaspoon, the other adds a cup. The soup is ruined, and no one knows who did it.
• Starvation: One chef is so busy that the other four never get a chance to cook.

The Gap: Existing tests for these robots only check if they can make a sandwich (sequential code). They don't test if the robots can manage a chaotic kitchen (concurrent code). A robot might write code that looks perfect on paper but causes a kitchen disaster when actually running.

The Solution: CONCUR (The "Kitchen Simulator")

The authors of this paper created a new test called CONCUR. Think of it as a high-tech kitchen simulator designed specifically to see if these AI robots can handle the chaos of a multi-chef environment.

Here is how they built it:

1. The Menu (The Dataset): They took 43 classic "kitchen coordination" problems from a textbook (like "How to manage a shared pot of soup") and created 72 slightly different versions of them. This gives them 115 unique challenges to test the robots.
2. The Rules (The Prompt): They gave the robots very strict instructions: "You must use only standard tools (Java 8), you must have exactly 3 chefs working, and you must finish in 10 minutes."
3. The Simulator (The Evaluation): This is the most important part. Instead of just reading the code to see if it looks right, they ran the code through a super-powerful simulator called Java PathFinder (JPF).
   • Imagine a time-traveling inspector who runs the kitchen scenario thousands of times in a split second.
   • He tries every possible order in which the chefs could move: Chef A moves first, then B... or B moves first, then A...
   • If the code crashes, freezes, or messes up in any of these scenarios, the robot fails.

What They Found (The Results)

They tested 23 of the smartest AI robots on this new test. Here is what happened:

• The "Good" News: Many robots could write code that looked correct and didn't crash immediately.
• The "Bad" News: When the simulator ran the code thousands of times, most robots failed.
  • Many robots forgot to actually create multiple chefs (they wrote code for one chef doing everything, which isn't "concurrent").
  • Many robots created "deadlocks" where the chefs got stuck waiting for each other.
  • Many robots created "race conditions" where the data got corrupted.
• The "Fake" Score: They checked a popular scoring system called CodeBLEU. This system is like a spellchecker that compares the robot's code to a human's code. It checks if the words and grammar are similar.
  • The Shock: A robot could get a high score on CodeBLEU (looking very similar to a human) but still have a kitchen disaster (a deadlock) when running. The spellchecker couldn't see the chaos.

The Big Takeaway

This paper tells us that just because code looks right, doesn't mean it works right when multiple things happen at once.

• Old Way: Check if the code compiles and looks similar to human code. (Like checking if a sandwich recipe has the right words).
• New Way (CONCUR): Simulate the code running in a chaotic, multi-threaded environment to see if it actually survives the chaos. (Like actually cooking the meal with five chefs to see if the kitchen explodes).

The authors conclude that we need better tools to test AI on complex, multi-tasking jobs. We can't just trust the "spellcheckers" anymore; we need the "kitchen simulators" to ensure our AI isn't just writing pretty code, but code that actually works in the real, messy world.

Technical Summary

Here is a detailed technical summary of the paper "CONCUR: Benchmarking LLMs for Concurrent Code Generation."

1. Problem Statement

While Large Language Models (LLMs) have shown significant success in generating sequential code, existing benchmarks (e.g., HumanEval, MBPP) fail to evaluate their capabilities in concurrent code generation.

• The Gap: Concurrent programming introduces unique complexities such as nondeterministic thread scheduling, synchronization requirements, and specific bug types (deadlocks, race conditions, starvation) that do not exist in sequential code.
• Limitations of Current Metrics: Traditional evaluation methods rely on:
  • Static Similarity (e.g., CodeBLEU): These capture surface-level syntax overlap but often miss semantic correctness in concurrent logic.
  • Unit Testing: Standard unit tests cannot systematically explore the vast, nondeterministic state space of thread interleavings, leading to false positives where flawed concurrent code passes tests.
• Consequence: There is no reliable way to assess whether LLMs can generate safe, correct multithreaded programs, creating a critical blind spot in software engineering AI research.

2. Methodology: The CONCUR Benchmark

The authors propose CONCUR, the first benchmark dedicated to evaluating LLMs on concurrent code generation. It consists of a curated dataset and a rigorous automated evaluation framework.

A. Dataset Construction

• Source: 43 concurrency problems derived from a standard concurrency textbook [14].
• Augmentation: To prevent memorization and increase diversity, the authors generated 72 validated mutant variants using an LLM (Gemini), resulting in a total of 115 problem instances.
• Ground Truth: Since the textbook provided only snippets, the authors reconstructed complete, runnable Java 8 ground-truth implementations.
• Prompt Engineering: Prompts were structured to enforce:
  • Java 8 syntax (no third-party libraries).
  • Bounded thread counts and iterations to prevent state-space explosion during verification.
  • Single-file output constraints to facilitate code extraction.
• Bounding Strategy: To make model checking tractable, the authors explicitly bounded the number of threads and loop iterations in both the prompts and the ground truth. This preserves the concurrency pattern (e.g., producer-consumer) while limiting the state space size.

B. Evaluation Framework

The framework employs a two-stage verification process:

1. Compilation: All generated code is compiled using a standard Java 8 environment. Programs failing to compile are flagged (errors include syntax errors, missing imports, or illegal third-party dependencies).
2. Model Checking (Java PathFinder - JPF):
   • Instead of running the code once, the framework uses JPF, an explicit-state model checker, to exhaustively explore all possible thread interleavings within a defined bound.
   • Custom Listeners: The authors implemented specific JPF listeners to detect:
     • Deadlocks (DL): Cyclic lock dependencies.
     • Race Conditions (RC): Unsynchronized conflicting accesses to shared variables.
     • Starvation (SV): Threads perpetually denied resources.
     • Single Thread (ST): A critical check to ensure the model actually spawned multiple threads (a common failure mode where LLMs use thread-safe classes but run sequentially).
     • Uncaught Exceptions (UE) & Termination Errors (TE).
   • Pass Criteria: A program is considered correct only if it compiles successfully AND JPF reports zero violations across all explored interleavings.

3. Key Contributions

1. CONCUR Benchmark: A dataset of 115 concurrency problems with structured prompts and validated Java ground truths.
2. Automated Verification Framework: A novel pipeline integrating compilation with model checking (JPF) to rigorously verify concurrent correctness, moving beyond simple unit testing.
3. Comprehensive Evaluation: An assessment of 23 state-of-the-art LLMs (including proprietary APIs like GPT-5/Claude and open-source models like Llama/Qwen).
4. Metric Analysis: A demonstration that static metrics like CodeBLEU are insufficient for concurrent code evaluation.
5. Public Resources: Release of the dataset, tools, and a public leaderboard for ongoing community benchmarking.

4. Experimental Results

The authors evaluated 23 LLMs on the 115 problems using pass@1 and pass@3 settings.

• Performance Overview:

  • Top Performers: gpt-5 achieved the highest pass rate (77.39% at pass@1, 91.30% at pass@3), followed by claude-opus-4-1 (67.83% / 79.13%).
  • General Struggle: Even the best models failed on nearly 10-25% of tasks. Lower-tier models (e.g., mistral:7b) had pass rates as low as 2.61%.
  • Impact of Sampling: Increasing generations from 1 to 3 significantly improved pass rates for all models, with weaker models seeing the most dramatic relative gains (e.g., codellama:34b jumped from 6.09% to 20.00%).

• Error Analysis:

  • Compilation Errors: The most common failure was syntax errors (71%) and missing imports.
  • Concurrency Bugs: Among compilable code, Uncaught Exceptions were the most frequent JPF error. Race Conditions and Deadlocks were also prevalent.
  • Single-Threaded Execution (ST): A significant failure mode where models generated code using thread-safe data structures (e.g., ConcurrentHashMap) but failed to spawn multiple threads, effectively running sequentially. This accounted for a non-negligible portion of failures.
  • Starvation: Surprisingly rare in the results, likely due to the low complexity and bounded nature of the test cases.

• CodeBLEU vs. Correctness:

  • The study found a weak correlation between CodeBLEU scores and actual correctness. Models with moderate CodeBLEU scores often produced code with fatal concurrency bugs, while some correct programs had lower similarity scores. This confirms that static similarity metrics cannot replace dynamic verification for concurrent code.

• Validation: A manual audit of 115 "correct" programs confirmed a 92% precision for the automated framework, validating its reliability while highlighting that some functional specification violations (e.g., incorrect data flow) can still slip through.

5. Significance and Future Work

• Paradigm Shift: CONCUR establishes that evaluating LLMs for concurrent code requires formal methods (model checking) rather than just unit testing or static analysis.
• Safety Implications: The inability of current LLMs to reliably generate deadlock-free or race-free code highlights a critical safety gap for deploying AI in production software engineering.
• Future Directions: The authors plan to extend the benchmark to other programming languages and integrate more advanced path execution analysis frameworks to handle larger state spaces and more complex semantic violations.

In summary, CONCUR provides the first rigorous, formal-methods-based benchmark for concurrent code generation, revealing that while LLMs are improving, they still struggle significantly with the subtle and nondeterministic nature of multithreaded programming.