RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
RuBench is a repository-level agentic coding benchmark featuring 25 natively authored Russian task specifications from live open-source projects, designed to evaluate product-grade coding agents on real maintenance work while ensuring data contamination resistance and revealing critical insights into deployed system behaviors through rigorous statistical analysis and trajectory auditing.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are hiring a team of expert mechanics to fix a fleet of complex, custom-built race cars. Until now, most tests for these mechanics have been written in perfect, formal English, like a technical manual. But in the real world, customers don't speak like manuals; they speak like people. They say things like, "My car started shaking every time I hit the brakes on a rainy Tuesday," in their own native language.
RuBench is a new test designed to see if AI coding assistants can actually understand these messy, real-world customer requests when they are written in Russian, not English.
Here is a breakdown of how the test works, what happened, and the surprising twist the researchers found, using simple analogies.
1. The Test: A "Real-World" Repair Shop
Most previous tests gave AI agents a clean, English description of a bug. RuBench is different:
- The Cars: The test uses five real, popular open-source software projects (like
aiohttporLaravel). These aren't fake puzzles; they are live codebases used by thousands of developers. - The Requests: Instead of a formal bug report, the researchers wrote 25 new requests from scratch in Russian. They sound like a business owner complaining to a contractor: "The bot crashes when we make it an admin; fix it, but don't break the existing chats."
- The Mystery Box: The researchers hid the "answer key" (the actual code fix and the tests that prove it works). The AI has to figure out the fix on its own. Only the researchers have the key to check if the AI succeeded.
- Freshness: All the bugs were found in code updates made after the AI models were trained. This ensures the AI couldn't have just memorized the answer from its training data.
2. The Mechanics: Who Got the Job Done?
The researchers tested four different "mechanic teams" (combinations of software tools and AI models). They ran each team through the 25 repairs three times to see how consistent they were.
- The Star Performer: The team using Claude Code with the "Opus 4.8" model was the best. It successfully fixed about 79% of the problems.
- The Middle Pack: The "Sonnet 5" model did almost as well (75%), and the "GPT-5.5" model (via Codex CLI) fixed about 67%.
- The Struggling Mechanic: The "Haiku 4.5" model (a cheaper, faster version) only fixed about 53% of the tasks.
The Catch: Because there were only 25 tasks, the difference between the top three teams wasn't statistically "proven" to be real—it could just be luck. However, the gap between the top teams and the struggling Haiku model was huge and clear. The top teams solved a completely different class of problems than the bottom one.
3. The Big Surprise: The "Silent Substitution"
This is the most interesting part of the paper. The researchers also tested a fifth team using a brand-new model called Fable 5.
When they looked closely at the "black box" logs of what the AI was doing, they discovered a secret:
- On 20% of the tasks, the Fable 5 model didn't actually do the work.
- Instead, the software product (Claude Code) silently swapped Fable 5 out for the older, stronger Opus 4.8 model in the middle of the job.
- Why? Fable 5 has safety guards that are very strict. When it saw a task involving standard internet protocols (like fixing a web header), its safety guard got nervous and said, "I'm not allowed to touch this," and the system automatically handed the job to Opus 4.8 to finish it.
The Lesson: The researchers realized that when we test AI products, we aren't just testing the "brain" (the model); we are testing the "body" (the whole software package). If the software secretly swaps the brain mid-task, the test results are lying about what that specific brain can do.
4. The Verdict
- Success: Product-grade AI agents are already good enough to handle real maintenance work specified in a non-English language (Russian). The best setup solved nearly 8 out of 10 tasks.
- Reality Check: The "cheap" models (like Haiku) are still significantly weaker, solving only about half the problems.
- Methodology: The paper proves that to get honest results, we must watch the AI's entire "diary" (trajectory) to make sure it didn't cheat by swapping models or looking up the answers online.
In short, RuBench shows that AI is getting good at speaking "human" (in Russian), but it also revealed that the software wrapping these AIs can sometimes be sneaky, swapping out the worker for a better one without telling anyone.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.