💻 computer science

APFuzz: Towards Automatic Greybox Protocol Fuzzing

This paper presents APFuzz, an automatic greybox protocol fuzzer that enhances testing effectiveness by employing static and dynamic analysis to infer accurate state models and leveraging Large Language Models for structure-aware, field-level message mutations.

Original authors: Yu Wang, Yang Xiang, Chandra Thapa, Hajime Suzuki

Published 2026-02-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yu Wang, Yang Xiang, Chandra Thapa, Hajime Suzuki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a hidden treasure inside a massive, locked castle (the computer program). The castle has many rooms (states), and the doors only open if you knock in a very specific rhythm or say a secret password (protocol messages).

Most security testers (fuzzers) are like people throwing thousands of random pebbles at the castle walls, hoping one hits a weak spot. This works for simple huts, but for complex castles with locked doors, it's inefficient. You might throw a pebble at the wrong door, or knock in the wrong rhythm, and the guard (the server) just ignores you.

APFuzz is a new, super-smart treasure hunter designed to solve this problem. It combines two superpowers to find bugs (the "treasure") faster and deeper than anyone else.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Black Box" Confusion

Traditional testers don't know how the castle works. They don't know:

The State: They don't know if the guard is currently "sleeping," "awake," or "angry." If you try to open a door while the guard is sleeping, nothing happens. You need to wake him up first with a specific sequence of knocks.
The Message: They don't know the exact shape of the key. If they try to jam a square peg into a round hole, the guard rejects it immediately.

2. Superpower #1: The "Mind Reader" (State Representation Learning)

To find the treasure, you need to know which room you are in.

Old Way: Testers had to ask a human expert, "Hey, how do I know if the guard is awake?" The expert would write a manual, but humans make mistakes, and writing these manuals is slow and boring.
APFuzz's Way: APFuzz is like a detective who reads the castle's blueprints (the source code) and watches the guards in action.
- Step 1 (Static Analysis): It scans the blueprints to find variables that might be the "mood switches" (e.g., a variable named is_logged_in).
- Step 2 (Dynamic Analysis): It watches the guards move around. If a variable changes values frequently and seems to control the doors, it keeps it. If a variable is just a random counter that doesn't matter, it throws it away.
- Result: APFuzz builds a perfect map of the castle's rooms automatically, without needing a human to draw it. It knows exactly when the guard is "awake" and ready for a new challenge.

3. Superpower #2: The "Translator" (Input Structure Learning with LLMs)

Once you know the room, you need the right key.

Old Way: Testers used to guess the shape of the key. They would randomly chop off pieces or change colors. For text-based castles (like HTTP), this was okay. But for binary castles (like 5G or encrypted data), the keys are complex blocks of code. Randomly changing them usually breaks the key, and the guard rejects it instantly.
APFuzz's Way: APFuzz uses a Large Language Model (LLM)—think of it as an AI that has read every manual ever written about these castles.
- You show the AI a sample key (a seed message).
- The AI says, "Ah, I see! This part is the 'Header,' this part is the 'Length,' and this part is the 'Secret Code.'"
- Now, instead of randomly smashing the key, APFuzz knows exactly which part to tweak. It can change the "Secret Code" without breaking the "Header."
- Analogy: Imagine a lock with 10 dials. Old testers spin all 10 dials randomly. APFuzz asks the AI, "Which dial controls the lock?" and then only spins that one. This saves time and gets you into the deep rooms faster.

4. The Result: Finding the Treasure Faster

The researchers tested APFuzz against the best existing treasure hunters (like AFLNET) using a standard set of 13 different "castles" (network protocols like SSH, DNS, and FTP).

Coverage: APFuzz explored 10% more rooms than the baseline. It didn't just knock on the front door; it found the secret passages in the basement.
Speed: It was 5 to 6 times faster at sending messages and checking rooms.
Bugs Found: It found the most "treasures" (crashes and security bugs). For example, in one test (TinyDTLS), it found 5 unique bugs while others found 4 or fewer. In another (Dnsmasq), it found a critical bug in 4,000 seconds, while others took over 24 hours or failed to find it at all.

Summary

APFuzz is like upgrading from a blindfolded person throwing darts to a Sherlock Holmes with a high-tech map and a translator.

It automatically learns the rules of the game (the state) by reading the code and watching the program run.
It understands the language of the messages (the structure) using AI, so it doesn't waste time sending broken keys.
Because it is smarter, it finds the hidden, dangerous bugs in our digital infrastructure much faster and more reliably than before.

This is a big deal because it means our digital systems (like 5G networks, banking, and the internet) can be tested more thoroughly and automatically, keeping us safer from hackers.

1. Problem Statement

Greybox protocol fuzzing aims to uncover vulnerabilities in stateful network protocol implementations by generating inputs based on code coverage and state coverage feedback. However, existing tools face two critical limitations:

State Representation Challenges: Accurately identifying protocol states is difficult. Existing tools rely on manual configuration (e.g., extracting HTTP status codes) or heuristic-based static analysis (e.g., finding enumerated variables), which often results in false positives/negatives and requires significant human intervention.
Input Structure Awareness: Traditional fuzzers use byte-level or message-level mutations. While effective for text-based protocols, these random mutations often break the strict syntax of binary protocols, causing the server to reject inputs early. While Large Language Models (LLMs) have been used to understand text-based protocol grammars, they have not been effectively applied to the field-level structure of binary protocols.

2. Methodology: APFuzz

The authors propose APFuzz, a coverage-based greybox fuzzer that enhances "smartness" through two novel automated learning modules: State Representation Learning and Input Structure Learning.

A. System Framework

APFuzz operates in a standard fuzzing loop but integrates two pre-processing and runtime learning stages:

State Representation Learning: Automatically identifies state variables in the source code to infer an accurate state model.
Input Structure Learning: Uses an LLM to parse initial seed messages and derive field-level grammar for binary protocols.

B. Key Technical Components

1. Automatic State Representation Learning (Two-Stage Approach)
To infer the state model without manual annotation, APFuzz employs a two-stage process to identify state variables:

Stage 1: Static Analysis: The system scans the source code to identify candidate state variables. It combines heuristics from existing tools (NSFuzz and SGFuzz), looking for:
- Global integers or struct members loaded/stored within network event loops.
- Enumerated types or variables assigned constant values (e.g., INIT, READY).
- Variables assigned enumerated values or named constants.
Stage 2: Dynamic Analysis: To filter out false positives (noise), the system instruments the code using LLVM and runs a lightweight dynamic analysis. It applies filtering rules:
- The variable must have a unique value count between a minimum ($min$) and maximum ($max$) threshold (e.g., 3–9 values).
- The hit count for specific values must exceed a threshold.
- Variables with names containing "state" are prioritized.
Outcome: This results in an accurate set of state variables used to track state sequences during fuzzing and update the state machine model.

2. Input Structure Learning via LLM
To handle binary protocols, APFuzz leverages an LLM to understand message structures:

Seed Interpretation: Initial seed messages (binary) are converted to binary text strings and fed to the LLM.
Prompt Engineering: Using in-context few-shot learning, the LLM is prompted to identify the field name, bit start position, and bit length for each part of the message.
Field-Based Mutation: Unlike byte-level mutations, APFuzz uses the LLM-derived structure to perform field-level mutations.
- The fuzzer selects a specific field range $[start, start+length]$.
- It applies deterministic or stack-based mutations (e.g., bit flips, swapping, overwriting with dictionary values) specifically within that field.
- A probability parameter ( $\epsilon$ ) determines whether to use field-based mutations or classical random mutations.

3. Key Contributions

Automated State Representation: A novel two-stage (static + dynamic) method to automatically identify state variables in source code, eliminating the need for manual annotation or response-code extraction scripts.
LLM-Driven Binary Protocol Fuzzing: The first approach to extend LLM-based grammar learning to binary protocols, enabling field-level, structure-aware mutations that respect protocol semantics.
Comprehensive Evaluation: A rigorous evaluation against the ProFuzzBench benchmark, comparing APFuzz with the baseline (AFLNET) and state-of-the-art tools (StateAFL, NSFuzz, ChatAFL, HNPFuzzer).

4. Experimental Results

The evaluation was conducted on 13 network servers (7 text-based, 6 binary) over 24-hour runs.

Code Coverage: APFuzz achieved the highest average coverage improvements among all tools.
- Branch Coverage: +10.03% improvement over the baseline AFLNET.
- Line Coverage: +8.42% improvement over AFLNET.
Throughput (Efficiency):
- APFuzz demonstrated high efficiency, achieving 5.19× higher execution throughput and 6.20× higher message-sending throughput compared to AFLNET.
- It outperformed specialized tools like HNPFuzzer in message throughput for many targets.
Vulnerability Discovery:
- APFuzz discovered the highest number of unique crashes (13) across all targets, outperforming NSFuzz (12) and StateAFL (12).
- Time-to-Exploit: APFuzz found critical bugs significantly faster. For example, it detected a heap-buffer-overflow in Dnsmasq in ~4,200 seconds, whereas other tools took >24 hours or failed to find it.
State Model Inference: APFuzz inferred more accurate and compact state models (fewer vertices/edges) compared to tools relying on response codes or memory hashes, leading to more efficient state space exploration.

5. Significance and Limitations

Significance:

Automation: APFuzz significantly reduces the human effort required to set up protocol fuzzers by automating both state variable identification and message structure parsing.
Binary Protocol Support: It bridges a critical gap in fuzzing research by applying LLMs to binary protocols, which are prevalent in modern systems (e.g., 5G/6G, IoT) but difficult to fuzz due to strict formatting.
Performance: It proves that "smarter" fuzzing (structure-aware and state-aware) does not come at the cost of speed; APFuzz is both more effective and more efficient than current state-of-the-art tools.

Limitations & Future Work:

Dependency Handling: The current implementation struggles with complex dependencies between fields in binary protocols (e.g., a length field depending on a previous field's value).
Filtering Noise: The dynamic analysis filtering relies on simple counting; advanced reverse engineering techniques could further improve the precision of state variable selection.
LLM Accuracy: While LLM parsing is generally accurate, errors in field identification can occur, though the system degrades gracefully to random mutation if parsing fails.

In conclusion, APFuzz represents a significant step forward in automated security testing, demonstrating that combining static/dynamic analysis with LLMs can effectively tackle the complexities of modern stateful network protocols.