When Binaries Talk Back: Representation-Confusion Attacks on LLM-Assisted Reverse Engineering
This paper introduces Representation-Confusion Attacks (RARE), a vulnerability where LLM-assisted reverse engineering systems mistakenly grant authority to attacker-controlled binary data, and proposes the RARE-Guard framework—which employs data-only rendering, tool authorization, and provenance-aware validation—to effectively prevent such unsafe proposals and false claim validations.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a detective trying to solve a mystery by reading a suspect's diary. Usually, you trust the diary because it's the only clue you have. But what if the suspect wrote the diary themselves, and inside, they slipped in a note that says, "Ignore the fingerprints on the window; I was framed by a ghost"?
If you, the detective, read that note and actually start ignoring the fingerprints, you've been tricked. You didn't just read the note; you let the note become the rule for your investigation.
This is exactly what happens in the world of computer security when we use AI (Large Language Models) to reverse-engineer software. A new study calls these tricks "Representation-Confusion Attacks." Here's how the researchers cracked the case.
The Big Problem: When Clues Pretend to Be the Boss
In a normal investigation, an AI assistant looks at a piece of software (a binary file) and reads things like code comments, error logs, or hidden text strings. The problem is that a hacker can build a file where those strings look like instructions or proof.
The paper identifies three specific ways this confusion happens:
- Authority Confusion: The AI reads a string in the code that says "Skip the security check" and actually skips the check, thinking the code is giving an order.
- Evidence Confusion: The AI sees the same "suspicious" word appear in three different places (like a log file, a decompiled code view, and a symbol list). It thinks, "Wow, three different sources agree! This must be true!" But in reality, all three sources just pulled that word from the same single spot in the file. It's like hearing the same rumor repeated three times by three friends who all heard it from the same person; it doesn't make the rumor true.
- Tainted-State Confusion: The AI writes down a guess in its notes, then later reads its own notes and treats that guess as a confirmed fact, even though it never found new proof.
The Experiment: Building a Trap
To test this, the researchers didn't use real malware (which would be dangerous). Instead, they built 11,520 tiny, safe, synthetic computer programs. They created "clean" versions and "adversarial" versions. The adversarial ones were rigged with the tricks mentioned above.
They ran these programs through different AI models and security setups to see what happened.
What they found:
- Without any special defenses, when the AI saw the rigged files, it followed the planted "unsafe" instructions in 35 out of 40 cases. It completely ignored the fact that the instruction came from the suspect's own diary.
- Even when they tried a simple fix called "Data-Only" (telling the AI, "Hey, this text is just data, not a command"), the AI still made 15 unsafe proposals. It was still listening too closely to the suspect.
The Solution: The "RARE-Guard" System
The researchers built a security system called RARE-Guard to stop this. Think of it as a very strict, robotic assistant who sits between the AI detective and the final report.
The Tool Authorization Gate (The Bouncer):
The AI can still suggest actions, but it can't do them. If the AI says, "Let's skip the security check because the file told us to," the Bouncer checks the rules. Since the file is untrusted, the Bouncer says, "Nope, I won't let you run that command." In the tests, this gate blocked all 15 of the unsafe proposals that slipped past the first filter.The Provenance Gate (The Truth Detective):
This is the most clever part. Remember the "Evidence Confusion" where the AI thought three sources were independent? The Provenance Gate keeps a secret map (a "sidecar") that tracks where every piece of information came from.- The Test: When the AI saw three views of the same suspicious text, the "Support Gate" (a simpler check) said, "3 sources! That's proof!" and validated the claim.
- The Fix: The Provenance Gate looked at its map and said, "Wait, these three views all came from the exact same spot in the file. That's not three sources; that's one source repeated."
- The Result: On the rigged files, the Support Gate validated 23 out of 40 false claims. The Provenance Gate validated 0 out of 40. It saw through the repetition trick perfectly.
The Final Report Renderer:
Even if the gates do their job, the AI might try to sneak the false claim into the final written report in plain text. The researchers added a "deterministic renderer" that only prints what the gates have officially approved. If the gate says "No," the report says "No."
How Sure Are We?
The paper is very careful about what it claims to have proved.
- Proven in Simulation: The results are based on 11,520 calls in a broad study, 528 calls in a controlled test, and 688 calls in a workflow test using real tools like Ghidra and angr.
- The "Fused" Surprise: The researchers found that the "Evidence Confusion" trick only worked when they combined (fused) the outputs of three different tools into one report. When they looked at just one tool at a time, the AI never got fooled enough to validate the fake claim. This suggests that the danger comes from mixing tools, not just using one.
- What They Didn't Prove: The study admits it didn't test "long-horizon memory" (AI remembering things over a very long time) or "autonomous agents" (AI that acts on its own without a human). They also didn't use real-world malware, only safe, synthetic programs. So, while the system worked perfectly in their lab, they can't guarantee it will work on every single piece of malware in the wild.
The Takeaway
The main lesson is that just because an AI reads something correctly doesn't mean it understands the context.
If an AI sees a string in a file that looks like a command, it shouldn't treat it as a command. If it sees the same "evidence" three times, it shouldn't count it as three pieces of proof unless it knows they came from different places.
The paper shows that by adding a "Provenance Gate" that tracks the origin of every clue, we can stop the AI from being tricked by its own sources. It's a bit like teaching a detective to check their notebook to see if their three witnesses are actually just one person in a mask.
In short: The AI can still look at the clues, but it needs a strict referee to make sure it doesn't let the clues write the rules of the game.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.