💻 computer science

Representation Matters: An Empirical Study of Program Representations for LLM Vulnerability Reasoning

This paper introduces RepBench, an empirical study demonstrating that static-analysis-based structural representations (specifically AST+PDG) significantly outperform raw source code in LLM vulnerability detection by mitigating context dilution and offering a superior accuracy-overhead tradeoff.

Original authors: Andrew Stoltman, Johnathan Tang, Haipeng Cai

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Andrew Stoltman, Johnathan Tang, Haipeng Cai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly distracted, detective (the AI) how to spot a specific type of crime in a massive library of books (the computer code).

For a long time, researchers assumed the best way to help the detective was to hand them the entire raw book. The logic was: "The more pages the detective reads, the more clues they will find."

This paper, titled "Representation Matters," challenges that idea. The authors built a testing ground called RepBench to see if giving the detective a condensed, structured summary of the book works better than giving them the raw text.

Here is the breakdown of their findings using simple analogies:

1. The Experiment: Raw Text vs. The "Blueprint"

The researchers took 107 real-world examples of code vulnerabilities (like a hidden trap in a building). They asked the AI to find these traps using three different types of "input":

Raw Source Code: The full, unedited book with all the words, comments, and formatting.
Graphs (AST, CFG, PDG): These are like architectural blueprints or flowcharts. They strip away the words and show only the structure: how the rooms connect, where the doors are, and how water (data) flows through the pipes.
Hybrids: A mix of the raw book and the blueprints.

2. The Big Surprise: Less is More

The results were counter-intuitive.

The Raw Book Failed: When the AI read the raw code, it got the answer right only 53.5% of the time. It was like trying to find a specific leak in a house by reading the entire encyclopedia of plumbing theory; the AI got lost in the noise.
The Blueprints Won: When the AI looked at the structured graphs (specifically a combination of a "Syntax Tree" and a "Dependence Graph"), it got the answer right 83.2% of the time.
The Hybrid Backfired: When the researchers gave the AI both the raw book and the blueprints, performance actually dropped compared to just the blueprints.

3. The "Context Dilution" Effect

Why did adding more information make the AI worse? The authors call this "Context Dilution."

Think of it like trying to find a needle in a haystack.

The Blueprint is just the needle. It's small, focused, and easy to see.
The Raw Code is the haystack.
The Hybrid is the haystack plus the needle.

The study found that when you give the AI the whole haystack (raw code) along with the needle (graphs), the AI gets distracted by the hay. It starts looking at irrelevant details—like a comment in the code or a helper function that has nothing to do with the bug. The "needle" gets lost in the "hay," and the AI misses the vulnerability.

4. The Efficiency Bonus

There was another benefit: Cost and Speed.

The raw code prompts were huge (like a 500-page novel).
The graph-only prompts were much smaller (like a 100-page summary).
Despite being smaller and cheaper to run, the graph prompts were more accurate.

5. What This Means for the Future

The paper concludes that for AI to be good at spotting security holes, we shouldn't just dump raw code into it. Instead, we should use static analysis tools (the tools that create the blueprints) to act as a "translator."

The Analogy:
Instead of asking the AI to read a 1,000-page legal contract to find a loophole, we should ask a lawyer to read it first, summarize the critical clauses into a 2-page memo, and then give that memo to the AI. The AI can then focus on the logic without getting distracted by the fluff.

In short: The paper proves that for AI security reasoning, structured, compact evidence is far superior to raw, unfiltered data. Giving the AI "more" information often just makes it "less" effective.

Technical Summary: Representation Matters for LLM Vulnerability Reasoning

Problem Statement

Large Language Models (LLMs) are increasingly deployed for automated vulnerability detection, yet the optimal method for representing program structure and semantics for LLM-based reasoning remains unclear. Current pipelines predominantly rely on raw source code as the primary input, operating under the implicit assumption that providing more source-level context yields better evidence for vulnerability reasoning.

This paper challenges that assumption. Vulnerability reasoning often depends on semantic relations—such as data propagation, control-flow guards, and pointer aliases—that are not explicit in surface syntax. While static analysis representations (e.g., Control-Flow Graphs, Program Dependence Graphs) are designed to expose these relations, their efficacy when used as prompts for LLMs has not been systematically benchmarked against raw source code. The core problem is determining whether structured, static-analysis-based representations can improve LLM reasoning accuracy and efficiency compared to raw source inputs, and whether combining them yields synergistic benefits or introduces noise.

Methodology

The authors introduce RepBench, a unified benchmark pipeline designed to evaluate how different program representations affect LLM vulnerability reasoning under a fixed task.

Dataset and Scope

Corpus: The study utilizes a curated subset of 107 real-world C/C++ testcases derived from the PrimeVul dataset, covering five Common Weakness Enumeration (CWE) categories:
- Arithmetic vulnerabilities: CWE-122 (Heap-based Buffer Overflow), CWE-190 (Integer Overflow), CWE-191 (Integer Underflow).
- Memory-lifecycle vulnerabilities: CWE-415 (Double Free), CWE-416 (Use After Free).
Auxiliary Track: An additional 19 cases using Enriched Program Dependence Graphs (ePDGs) generated via the VulChecker/Hector pipeline, which includes LLVM-level metadata.

Experimental Design

The benchmark evaluates ten representation variants across three prompt families:

Source-only: Raw source code (Baseline).
Graph-only: Serialized structural representations including Abstract Syntax Trees (AST), Control-Flow Graphs (CFG), Program Dependence Graphs (PDG), and their combinations (AST+CFG, AST+PDG, CFG+PDG).
Source-plus-Graph: Hybrids combining raw source code with one or more graph representations.

Key Controls:

Task: Fixed binary classification (VULNERABLE vs. SAFE) for a target function given a specific CWE.
Prompting: All variants use a fixed Chain-of-Thought (CoT) prompting strategy with structured JSON output and dynamic few-shot selection.
Model: Experiments utilize qwen3.6-plus with a fixed temperature ( $T=0.2$ ).
Serialization: Graph artifacts (DOT/JSONL) are converted into compact, line-oriented text to minimize prompt overhead while preserving structural links.

Metrics

Curated Accuracy: Correctness on retained rows with valid binary predictions (isolating representation quality from parsing failures).
Effective Accuracy: End-to-end success rate including parsing failures.
Prompt Overhead: Measured by character count and input token count.

Key Contributions

RepBench Framework: A benchmark framework that transforms real-world vulnerability testcases into multiple program representations (AST, CFG, PDG, ePDG) and evaluates them under a standardized LLM detection task.
Systematic Ablation Study: A comprehensive evaluation of raw source, individual graphs, graph combinations, and source-plus-graph hybrids, isolating representation as the sole experimental variable.
Discovery of Context Dilution: Identification and characterization of a "context dilution effect," where adding raw source code to compact structural evidence degrades reasoning performance rather than improving it.
Optimal Representation Identification: Empirical evidence that AST+PDG is the most effective representation, significantly outperforming raw source and other variants.

Results

1. Graph Representations Outperform Raw Source

Contrary to the assumption that more source context is better, graph-only prompts achieved 74.4% curated accuracy, a 20.9 percentage point (pp) improvement over the raw-source baseline (53.5%). This demonstrates that compact structural evidence can substitute for and outperform direct source-code prompting.

2. AST+PDG is the Optimal Variant

The AST+PDG combination achieved the highest accuracy at 83.2% (a 29.7 pp improvement over raw source).

AST provides local syntactic anchors (operators, calls, declarations).
PDG provides cross-statement dependency evidence (data flow, control guards).
AST+CFG followed closely at 80.4%, while CFG+PDG scored 73.8%.
Source-plus-graph hybrids (e.g., pdg_plus_source, full) consistently underperformed their graph-only counterparts (e.g., pdg_plus_source at 65.7% vs. pdg at 69.0%), indicating that raw source adds distracting noise rather than complementary value.

3. The Context Dilution Effect

The study identifies a context dilution effect: adding raw source code to structural graphs increases prompt size (2.4× more characters, 2.2× more tokens) but reduces accuracy.

Mechanism: Even when prompts do not exceed the context budget (no truncation), the additional source text causes the model to attend to irrelevant code (e.g., unrelated helper functions or interface routines), obscuring the vulnerability-relevant structural evidence.
Efficiency: Graph-only prompts achieve the highest accuracy with the lowest overhead (average 5,973 tokens vs. 12,938 for source-plus-graph).

4. Variation Across Vulnerability Types

Arithmetic Vulnerabilities (CWE-190): Graph-only prompts significantly outperformed source-plus-graph prompts (77.8% vs. 65.8%), suggesting focused structural evidence is critical for source-to-sink reasoning.
Memory-Lifecycle Vulnerabilities (CWE-415/416): Source-plus-graph prompts performed competitively or slightly better on CWE-415, suggesting raw context may occasionally aid in reasoning about cleanup orders and pointer lifetimes, though graph-only remains robust.
Label Asymmetry: The model was more reliable on SAFE cases (76.1% curated accuracy) than VULNERABLE cases (66.0%), likely due to the difficulty of reasoning about the absence of guards or subtle data flows.

5. Auxiliary ePDG Findings

The ePDG track (LLVM-level enriched metadata) achieved 57.9% curated accuracy. While lower than Joern-based variants, this suggests vulnerability-oriented metadata has potential, though current limitations in readability (abstract LLVM IR) and artifact generation feasibility constrain its utility.

Significance and Claims

The paper claims that representation choice is a primary driver of LLM vulnerability reasoning quality, often more so than the volume of input context.

Re-evaluating Current Practice: Systems relying solely on raw source prompting may fail to leverage decades of progress in static analysis. The assumption that "more context is better" is challenged; in security reasoning, focused structural evidence is superior to unfiltered source text.
Static Analysis as a Prompt Layer: The authors propose that static analysis should serve as a prompt-construction layer. By extracting compact, security-relevant structural evidence (like AST+PDG) and presenting it to LLMs, developers can achieve higher accuracy with lower inference costs and reduced risk of context dilution.
Empirical Foundation: This work provides the initial empirical foundation for designing representation-aware LLM vulnerability-analysis systems, moving beyond simple code-injection to structured, analysis-driven prompting.

The study concludes that while raw source code is natural for LLMs, it is not optimal for security reasoning. Carefully selected structural representations offer a stronger accuracy-overhead tradeoff, motivating a shift toward using static analysis to construct focused prompts for security-focused LLM reasoning.