💻 computer science

Adversarial SQL Injection Generation with LLM-Based Architectures

This paper introduces two novel LLM-based systems, RADAGAS and RefleXQLi, for generating adversarial SQL injection payloads and evaluates their effectiveness against diverse Web Application Firewalls, revealing that while these models significantly outperform baselines in bypassing AI/ML-based defenses, they struggle against rule-based systems and demonstrate that payload diversity does not always correlate with higher bypass success.

Original authors: Ali Karakoc, H. Birkan Yilmaz

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Karakoc, H. Birkan Yilmaz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The Digital Locksmith and the Master Key

Imagine the internet is a giant city of buildings (websites). To keep these buildings safe from thieves, security guards (called Web Application Firewalls or WAFs) stand at the front doors. Their job is to stop bad guys from sneaking in through the back door using a specific trick called SQL Injection (a way of tricking the building's database into revealing secrets).

For a long time, security experts have tried to test these locks by hiring human locksmiths or using robotic tools that try every known key shape. But the locks are getting smarter, and the old keys don't work as well anymore.

This paper asks a new question: What if we use a super-smart AI (a Large Language Model or LLM) to invent new keys that the security guards have never seen before?

The researchers built two new "AI Key-Makers" to see how good they are at tricking these security guards.

The Two New AI Key-Makers

The researchers created two different methods to help the AI generate these tricky keys:

1. RADAGAS: The "Library Researcher"

Think of this AI as a brilliant student who has read every book in the world's largest library of past heists.

How it works: Before trying to pick a lock, this AI goes to its "library" (a database of known successful attacks from OWASP, GitHub, etc.). It grabs the best, most proven tricks from the past.
The Twist: It uses a special technique called RAG (Retrieval Augmented Generation). Instead of just guessing, it looks up the most relevant "cheat codes" and then asks the AI to remix them into something new.
The Result: It's like a thief who studies the blueprints of every successful robbery before attempting a new one.

2. RefleXQLi: The "Critical Thinker"

This AI works like a two-person team: a Creator and a Critic.

How it works:
1. The Creator tries to design a new key.
2. The Critic (another AI) looks at the key and says, "No, that looks too suspicious. Try again."
3. The Creator listens, thinks harder (using a "Chain of Thought" process), and tries to make a better, sneakier key.
The Twist: They keep arguing back and forth until the Creator makes a key that the Critic can't spot. It's like a writer and an editor working together until the story is perfect.

The Great Test: 2.2 Million Attempts

To see who was the best, the researchers set up a massive simulation:

The Attackers: They used 7 different systems (including the two new ones, plus old-school tools and standard AI).
The Defenders: They tested against 10 different security guards (WAFs). Some were old-school rule-followers (like a guard with a checklist), some were AI-based (guards that learn by watching), and some were famous commercial guards (like Cloudflare and AWS).
The Scale: They generated 240,000 keys and tried them on the guards 2.2 million times.

What Did They Find?

1. The "Library Researcher" (RADAGAS) Won the Race

The RADAGAS system, especially when powered by the GPT-4o AI, was the most successful overall. It managed to sneak past the guards about 22.7% of the time.

Why? Because it didn't just guess; it learned from a curated list of things that actually worked in the past.

2. The "Critical Thinker" (RefleXQLi) Was Very Stable

The RefleXQLi system was slightly less successful overall (about 21.2%), but it was very consistent. It rarely made mistakes and always produced unique keys.

3. The "Diversity Myth" Was Debunked

A common belief in security is: "The more different your keys are from each other, the better your chances."

The Paper's Finding: This wasn't true! The researchers found no strong link between making thousands of totally different keys and actually breaking in.
The Analogy: It doesn't matter if you try 1,000 different shaped keys if none of them fit the lock. It's better to try 100 keys that are very similar to the one you know works, than 1,000 random shapes.
Surprise: Sometimes, trying to make the keys too different actually made them worse.

4. Different Guards Need Different Keys

There was no "magic key" that opened every door.

AI Guards (Machine Learning WAFs): The RADAGAS system was a monster against these, breaking in over 90% of the time. It seems these AI guards weren't trained well enough to spot the "remixed" old tricks.
Rule-Based Guards (Checklist WAFs): These were very tough. The new AI systems struggled here, getting blocked almost all the time. The old-school tools (like SQLmap) actually did slightly better against these specific guards.
Commercial Guards (Cloudflare/AWS): The results were mixed. Different AI systems found different "holes" in their defenses.

5. The "Temperature" Setting Matters

AI models have a "creativity dial" called Temperature.

GPT-4o worked best when set to be less creative (low temperature, more logical).
DeepSeek worked best when set to be very creative (high temperature).
Claude was happiest in the middle.
Lesson: You can't use the same settings for every AI; you have to tune them like a radio to get the best signal.

The Bottom Line

The paper concludes that if you want to test if a website is secure, you shouldn't just rely on one tool.

Use RADAGAS to test against AI-based security systems.
Use RefleXQLi (or seed-driven versions) for stable, high-quality testing.
Don't obsess over making your attack keys "diverse"; focus on making them smart and relevant to the specific lock you are trying to open.

The researchers emphasize that they did this in a safe, isolated lab to help improve security, not to help criminals. They kept the "keys" private so no one could misuse them.

Technical Summary: Adversarial SQL Injection Generation with LLM-Based Architectures

Problem Statement

SQL Injection (SQLi) remains a critical vulnerability within the OWASP Top 10, yet traditional defense mechanisms and testing methodologies face significant limitations. Conventional approaches rely on manual penetration testing or automated rule-based tools like SQLmap. These methods struggle against modern defenses, particularly AI/ML-based Web Application Firewalls (WAFs) and commercial solutions, due to their deterministic nature and inability to generate novel, context-aware attacks. Furthermore, existing Large Language Model (LLM) research in security testing often suffers from a single-model focus, limited parameter exploration, and insufficient diversity mechanisms, leading to repetitive payloads that are easily detected by advanced defense systems.

Methodology

The authors introduce a comprehensive benchmarking framework to evaluate LLM-based SQLi generation against a diverse set of defenses. The study utilizes a testbed comprising seven generation systems, ten WAFs, and a MySQL execution validator.

System Architectures

The study proposes two novel architectures and compares them against three baselines:

RADAGAS (Retrieval Augmented Generation for Adversarial SQLi): This system integrates RAG with LLM generation. It retrieves proven attack patterns from a curated knowledge base (OWASP, PortSwigger, GitHub) using FAISS and Maximum Marginal Relevance (MMR) to balance relevance and diversity. The system employs multi-stage diversity filtering (semantic, lexical, and functional) and execution validation. Three variants were tested using different foundation models: GPT-4o, DeepSeek-R1, and Claude 3.7 Sonnet.
RefleXQLi (Reflective Chain-of-Thought SQLi): This architecture employs a four-step Chain-of-Thought (CoT) reasoning process (WAF Analysis, Strategy Formulation, Payload Design, Refinement) followed by a dual-LLM adversarial loop. A generator LLM creates payloads, while a discriminator LLM evaluates and rejects them based on malicious intent, forcing the generator to refine the payload iteratively until it bypasses the discriminator.
Baselines:
- Traditional: A simulated SQLMap engine using rule-based deterministic payloads.
- Vanilla GPT-4o: A zero-shot LLM without fine-tuning or specialized prompting.
- GenSQLi: A template-based approach using in-context learning.

Experimental Setup

Scale: 240 distinct experimental configurations generated 240,000 payloads, resulting in approximately 2.2 million tests.
Defenses: Tests were conducted against 10 WAFs:
- AI/ML-based: WAF-Brain (LSTM) and CNN-WAF.
- Rule-based (Open Source): ModSecurity (PL1–3) and Coraza (PL1–3).
- Commercial: AWS WAF and Cloudflare WAF.
Validation: A vulnerable web application backed by MySQL 8.0 served as a ground truth for execution validity.
Metrics: Performance was measured via bypass rates, execution success, and six diversity metrics (semantic, lexical, contextual, n-gram, AST, and functional diversity).

Key Results

Overall Performance

RADAGAS Superiority: The RADAGAS variants outperformed all baselines, with RADAGAS-GPT4o achieving the highest mean bypass rate of 22.73%.
RefleXQLi: Achieved a 21.21% bypass rate with the lowest variance ( $\sigma = 0.37$ ), indicating stable output.
Baselines: Traditional SQLi (15.01%) and Vanilla GPT-4o (12.90%) performed significantly worse. GenSQLi achieved 20.35%.

Model-Specific Optimization

The study found that optimal parameters are model-specific and counter-intuitive:

GPT-4o: Performed best at low temperature ( $T=0.1$ ) with a high diversity threshold ( $\theta=90$ ).
DeepSeek-R1: Performed best at high temperature ( $T=0.9$ ) with a high diversity threshold ( $\theta=90$ ).
Claude 3.7: Optimal at medium temperature ( $T=0.6$ ) with a lower diversity threshold ( $\theta=75$ ).

Defense-Specific Effectiveness

AI/ML WAFs: RADAGAS variants were highly effective, bypassing WAF-Brain at 92.49% (DeepSeek) and CNN-WAF at 80.48% (Claude).
Rule-Based WAFs: All LLM-based systems struggled against rule-based defenses (ModSecurity/Coraza), with bypass rates generally between 0% and 5.70%. GenSQLi achieved the highest rule-based bypass (16.26% on ModSecurity PL1), though many were identified as false positives or tautological expressions.
Commercial WAFs: RADAGAS-GPT4o achieved the highest bypass on Cloudflare (49.50%), while RefleXQLi performed best on AWS WAF (33.00%).

Diversity and Seed Analysis

Diversity Correlation: Contrary to the assumption that diversity drives success, the study found no statistically significant correlation between payload diversity metrics and bypass rates across the seven systems. In fact, less diverse payloads sometimes achieved higher bypass rates.
Seed Effect: An ablation study using "V2" seeds revealed that RefleXQLi V2 (CoT + Seeds) achieved a 35.78% bypass rate, significantly outperforming the raw seeds (28.93%) and GenSQLi V2 (29.84%). This suggests that high-quality seeds combined with CoT reasoning yield superior results compared to template mutation alone.
Execution Validity: RADAGAS systems achieved 60–64% MySQL execution success, whereas RefleXQLi (without seeds) only achieved 27.80%, highlighting a trade-off between creative adversarial generation and semantic correctness.

Significance and Claims

The paper claims to provide one of the largest benchmarking studies of LLM-based adversarial SQLi generation, offering a holistic view of the interaction between LLM architectures and modern defenses.

Architectural Insights: The study demonstrates that RAG-based approaches are highly effective against AI/ML-based WAFs by leveraging curated knowledge, while CoT-based reasoning provides stability and semantic correctness. No single system universally outperforms others; a combination of approaches is necessary for comprehensive coverage.
Parameter Sensitivity: The findings challenge the notion of universal parameter settings, showing that temperature and diversity thresholds must be empirically tuned for each specific foundation model.
Diversity Myth: The research challenges the prevailing assumption that payload diversity is the primary driver of WAF bypass success, suggesting that the quality of the initial context (seeds) and the reasoning mechanism (CoT/RAG) are more critical factors.
Practical Guidelines: The authors provide actionable guidelines for security testing, recommending the use of RAG for AI/ML WAFs and seed-driven CoT for maximum bypass rates, while acknowledging the limitations of current LLMs against strict rule-based systems.

The authors emphasize that these findings are specific to the tested configurations and time (early 2026) and call for further research into other attack types and database dialects. All experiments were conducted in isolated, authorized environments to ensure responsible disclosure.