🤖 AI

CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions

CodeHacker is an automated agent framework that enhances the evaluation of code generation models by employing a multi-strategy approach and a self-calibration phase to generate targeted adversarial test cases, effectively exposing vulnerabilities in competitive programming solutions and improving the robustness of both benchmark datasets and RL-trained models.

Original authors: Jingwei Shi, Xinxiang Yin, Jing Huang, Jinman Zhao, Shengyu Tao

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingwei Shi, Xinxiang Yin, Jing Huang, Jinman Zhao, Shengyu Tao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, high-stakes coding tournament where thousands of programmers (and now, AI models) submit solutions to tricky math and logic puzzles. In this world, there's a special rule: if you can find a specific, weird input that breaks someone else's code, you "hack" them and win points. This is called hacking, and it's the ultimate test of whether a solution is truly robust.

The paper introduces CodeHacker, an automated "super-hacker" designed to act like a relentless detective in this tournament. Its job isn't to solve the puzzles, but to break the solutions submitted by others (including AI models) to see if they are actually correct.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Fake Pass"

Currently, when we test AI code, we use a set of standard test cases (like a checklist). The paper argues that these checklists are often too easy. They miss the "weird edge cases"—the tricky, unusual scenarios where a program might fail.

Analogy: Imagine a car safety test where you only drive the car on a perfect, empty highway. The car passes! But you never tested it on a icy, winding mountain road. The car looks safe, but it's actually dangerous in real life. Existing benchmarks are like that perfect highway; they let "flawed" solutions pass because they never hit the potholes.

2. The Solution: The "CodeHacker" Agent

CodeHacker is an AI agent designed to be the ultimate "pothole finder." Instead of randomly guessing, it acts like a competitive programmer trying to break a specific piece of code. It uses three main strategies:

Stress Testing: It throws massive amounts of data at the code to see if it crashes or runs out of memory (like trying to fill a bucket with a firehose).
Logic Targeting: It reads the code, figures out the logic, and specifically crafts inputs to trick that logic (like finding the exact key that fits a lock).
Anti-Hash Attacks: Some coders use "hashes" (mathematical shortcuts) to check answers. CodeHacker has a special math trick to find two completely different inputs that produce the same hash result, tricking the code into thinking they are the same.

3. The "Calibration" Phase: Fixing the Referee

Before CodeHacker starts breaking things, it has to make sure the "referee" (the system that checks if an answer is right) is perfect.

The Problem: Sometimes the referee itself is broken. It might let a wrong answer pass (too loose) or reject a right answer (too strict).
The Fix: CodeHacker first tries to "hack" the referee itself. It generates tricky inputs to see if the referee makes a mistake. If the referee fails, CodeHacker fixes the referee's rules.
Analogy: Before a boxing match, the referee checks their own gloves and rules to make sure they won't accidentally knock out a boxer who didn't break the rules. CodeHacker ensures the referee is fair and accurate before the real fight begins.

4. The Results: Cleaning Up the Scoreboard

When the authors used CodeHacker on existing datasets, they found a lot of "False Positives."

What happened: Many solutions that were previously marked as "Correct" (Accepted) were actually broken. CodeHacker found the specific inputs that made them fail.
The Outcome: By filtering out these "lucky" but broken solutions, the evaluation became much more honest. It's like removing the players who only won because the referee missed a foul.

5. Training Better AIs

The paper also shows that training AI models using these "hacked" examples makes them smarter.

Analogy: If you only practice against easy opponents, you'll never learn to win a championship. But if you train against a coach who specifically shows you your weaknesses and how to break your own bad habits, you become a much stronger fighter.
Result: AI models trained on these difficult, adversarial examples performed significantly better on new, unseen challenges than those trained on standard, easy data.

Summary

CodeHacker is a tool that mimics the "hacking" phase of competitive programming to rigorously test code. It first fixes the testing tools to ensure they are accurate, then systematically hunts for hidden bugs in solutions that look correct but aren't. By doing this, it creates a much tougher, more reliable standard for judging how good AI code really is, ensuring that only truly robust solutions get the gold star.

Technical Summary: CodeHacker

Problem Statement

The evaluation of Large Language Models (LLMs) for code generation relies heavily on the quality and robustness of test cases. Existing benchmarks often suffer from insufficient coverage of subtle corner cases, allowing incorrect solutions to pass (False Positives). Current automated approaches largely function as "Black-box Fuzzers," relying on probabilistic input exploration or heuristic mutations. These methods often fail to target specific logical vulnerabilities of a concrete implementation, prioritizing statistical coverage over deep semantic understanding. Consequently, evaluation outcomes may hinge on whether test cases accidentally cover failure cases rather than on whether the model truly understands problem semantics and constraint structures.

Methodology

The authors propose CodeHacker, an automated agent framework designed to generate targeted adversarial test cases that expose latent vulnerabilities in program submissions. Mimicking the "hack" mechanism in competitive programming, the framework treats individual programs as first-class objects and systematically searches for failure-inducing counterexamples.

The framework operates in two primary phases:

Phase I: Evaluation Tool Calibration

Before attempting to hack contestant submissions, CodeHacker ensures the reliability of its "ammunition" (test cases) and "judgment" (checkers/validators) through an iterative refinement process:

Validator Refinement: The agent attacks the input validator to identify weaknesses. It performs Bypass Attacks (generating invalid inputs that the validator incorrectly accepts) and Rejection Attacks (generating valid edge cases that the validator incorrectly rejects). The validator is patched until it strictly aligns with problem definitions.
Checker Refinement: The agent probes the output checker for vulnerabilities. It constructs Deception Attacks (incorrect outputs that mimic valid answers) and Rejection Attacks (valid alternative outputs that a rigid checker rejects).
Anti-Hallucination Pipeline: To ensure the integrity of the checker update, a three-step verification pipeline is employed: restricting inputs to trivial boundary cases, requiring explicit Chain-of-Thought reasoning from the generator LLM, and cross-verifying results with an independent LLM-as-a-Judge.
Expert Intervention: For complex verification logic (e.g., proving graph Hamiltonian paths) that exceeds typical LLM capabilities, human experts intervene to manually patch the tools. This occurs in less than 5% of cases, primarily for high-difficulty problems.

Phase II: Adversarial Case Generation

Armed with calibrated tools, the agent generates adversarial inputs using a Code Analyst and three complementary generation strategies:

Code Analyst: Acts as a strategist using a Dual-Execution Interface. It combines Behavioral Probing (executing the target binary in a C++ sandbox to verify hypotheses about crashes or incorrect outputs) and Precise Calculation (using a Python interpreter to verify complexity, boundary conditions, and mathematical properties like integer overflows).
Stress Test Generator: Systematically explores input space boundaries (e.g., maximum $N$ , fully connected graphs) to expose Runtime Errors (RE), Time Limit Exceeded (TLE), and Memory Limit Exceeded (MLE).
LLM-Based Generator: Guided by the Code Analyst, this module crafts targeted semantic test cases to trigger specific failure modes (Wrong Answer, RE, TLE, MLE) based on identified logical vulnerabilities.
Anti-Hash Generator: Targets submissions using polynomial rolling hashes. For fixed parameters, it formulates collision search as a Shortest Vector Problem (SVP) and applies lattice reduction (LLL algorithm). For smaller moduli or non-linear hashes, it employs a Birthday Attack strategy.

Key Contributions

Formalization of Hacking: The authors formalize "hacking" as an adversarial test generation task and design an LLM-driven agent that actively searches for high-value corner cases and logical counterexamples, moving beyond traditional mutation-based methods.
Improved Evaluation Metrics: By deploying the agent on existing code datasets, the framework demonstrates substantial improvements in both the True Negative Rate (TNR) and True Positive Rate (TPR), indicating that generated adversarial cases meaningfully strengthen evaluation robustness.
CodeHackerBench: The authors introduce a new evaluation setting, CodeHackerBench, constructed by mining judgment discrepancies in the CodeContest+ dataset. This benchmark is designed to better characterize models' abilities to reason about incorrect code and extreme failure scenarios.

Results

Experiments conducted on 2,000 problems from the CodeContest+ dataset and evaluations on LiveCodeBench yield the following findings:

Evaluation Rigor: CodeHacker significantly improves the TNR of existing datasets. For Special Judge problems, the TNR increased from 82.18% (baseline) to 96.05% after adding adversarial hack cases. This corrects "metric inflation" where flawed solutions previously passed due to weak test coverage.
Reasoning Capability: The Hack Success Rate (HSR) serves as a strong differentiator for advanced reasoning models. For instance, DeepSeek V3.2 achieved an HSR of 64.83%, while its performance dropped to 23.76% when explicit reasoning was disabled, confirming that generating valid hacks is a reasoning task requiring constructive derivation of algorithms.
RL Training Efficiency: Models trained on the augmented CodeContests++ dataset (containing adversarial cases) showed consistent performance improvements on the out-of-distribution LiveCodeBench benchmark compared to models trained on standard data. This suggests that adversarial inputs serve as dense reward signals, fostering genuine algorithmic reasoning rather than pattern matching.
Rank Reversal: The rigorous adversarial evaluation revealed rank reversals; for example, GPT-5-Mini was found to be more robust than Gemini-3.0-Flash when evaluated against the refined benchmarks, a distinction not visible in original tests.

Significance and Claims

The paper claims that CodeHacker establishes adversarial generation as a critical aspect for evaluating general algorithmic intelligence. The authors argue that the ability to generate valid hacks reflects a model's deep understanding of program behavior and provides a more discriminative signal for assessing advanced program reasoning capabilities than functional correctness alone.

Beyond academic evaluation, the authors envision CodeHacker as a vital component of next-generation competitive programming infrastructure, assisting problem setters in quality assurance and contestants in personalized training. They further posit that the core principle—systematic regression-test augmentation to probe untested boundary conditions—is conceptually transferable to real-world issue-solving benchmarks (e.g., SWE-bench) and broader repository-level evaluations to improve evaluation rigor in software engineering tasks.

The work adheres to strict ethical principles, prohibiting unauthorized data scraping and mandating that all adversarial evaluations be conducted in local, sandboxed environments to prevent Denial of Service (DoS) risks to public Online Judges.