Do Code LLMs Do Static Analysis?

1. Problem Statement

The paper addresses a fundamental question in the field of software engineering and artificial intelligence: Do Large Language Models (LLMs) perform static analysis as part of their reasoning process when solving code intelligence tasks?

• Context: Human programmers rely heavily on static analysis (examining code without execution) to build mental models of program behavior. This includes understanding function call relationships (call graphs), data flow between variables, and hierarchical code structures (Abstract Syntax Trees or ASTs).
• Hypothesis: If Code LLMs function similarly to humans, they should either:
  1. Naturally perform static analysis when tasked with code generation, summarization, or translation.
  2. Show improved performance on code tasks if explicitly trained on static analysis tasks.
  3. Acquire static analysis capabilities as a byproduct of training on code intelligence tasks.
• Gap: While LLMs are often anthropomorphized as "reasoning," there is insufficient empirical evidence regarding whether they actually utilize static analysis mechanisms or merely pattern-match syntactic forms.

2. Methodology

The authors conducted a comprehensive empirical study involving four models (two open-source: Jam, CodeLlama; two closed-source: GPT-4o, Gemini) across two programming languages (Java, C/C++).

Experimental Tasks

The study evaluated three Static Analysis Tasks and three Code Intelligence Tasks:

1. Static Analysis Tasks:
   • AST Generation: Converting source code to XML-based Abstract Syntax Trees.
   • Call Graph Generation: Identifying function invocation relationships.
   • Data Flow Graph Generation: Tracing variable usage and propagation.
2. Code Intelligence Tasks:
   • Code Summarization: Generating natural language descriptions of code.
   • Code Translation: Translating code between languages (e.g., Java to C#, C to Rust).
   • Code Generation: Generating code from natural language prompts.

Research Questions (RQs) & Experimental Design

The study was structured around five specific Research Questions:

• RQ1 (Baseline Performance): How do closed-source LLMs perform on static analysis tasks without fine-tuning?
  • Method: Evaluated GPT-4o and Gemini using zero-shot and in-context learning (with and without "Chain of Thought" prompts).
• RQ2 (Fine-tuning Static Analysis): Can open-source models learn static analysis tasks when fine-tuned?
  • Method: Fine-tuned Jam and CodeLlama on AST, Call Graph, and Data Flow datasets.
• RQ3 (Transfer to Code Tasks): Does fine-tuning on static analysis improve performance on code intelligence tasks?
  • Method: Fine-tuned models on static analysis first, then fine-tuned on code summarization, translation, and generation.
• RQ4 (Transfer from Code Tasks): Does training on code intelligence tasks result in static analysis capabilities?
  • Method: Fine-tuned models on code tasks, then evaluated their ability to generate static analysis artifacts.
• RQ5 (Error Analysis): What are the specific failure modes of LLMs in static analysis?
  • Method: Categorized errors (e.g., missing/extra edges, cycles, tag mismatches) based on established taxonomies.

Datasets and Metrics

• Datasets: Used standard benchmarks including 170k (Java), CodeXGLUE, CONCODE, Xlcost, and custom datasets derived from funcom and Haque et al.
• Metrics:
  • Static Analysis: Levenshtein distance (AST), Jaccard similarity, Pair Accuracy, Chain Accuracy (Call/Data Flow).
  • Code Tasks: METEOR, USE, BLEU, CodeBERTScore, Pass@k, Compilation Rate.

3. Key Results

A. Performance on Static Analysis (RQ1 & RQ2)

• Poor Baseline Performance: Closed-source models (GPT-4o, Gemini) performed poorly on complex static analysis tasks (Call Graphs, Data Flow) even with in-context learning.
  • AST: In-context learning helped significantly (e.g., Gemini improved from 0.64 to 0.93 Levenshtein score in Java), suggesting LLMs can learn syntax (tags) easily.
  • Call/Data Flow: In-context learning provided negligible or no improvement. Chain accuracy remained near zero, indicating a failure to understand global program structure.
• Fine-tuning Effects: Fine-tuning open-source models improved AST generation and partially improved Call/Data Flow metrics (Pair Accuracy). However, Chain Accuracy (the ability to get the entire graph correct) remained very low, even for fine-tuned models.

B. Impact on Code Intelligence Tasks (RQ3)

• No Generalization: Fine-tuning models on static analysis tasks did not significantly improve their performance on code summarization, translation, or generation.
• Logical Errors Persist: Models fine-tuned on static analysis still made the same logical errors as baseline models. For example, they continued to generate code with incorrect logic or parameters, suggesting they did not internalize the "semantics" of the static analysis tasks.

C. Static Analysis as a Byproduct (RQ4)

• No Emergent Ability: Training models on code intelligence tasks (summarization, generation) did not result in the ability to perform static analysis.
• Performance Drop: When asked to generate Call Graphs or ASTs after being trained on code tasks, the models performed significantly worse than models specifically fine-tuned for those tasks.

D. Error Analysis (RQ5)

• Syntactic vs. Semantic: The most common errors were semantic hallucinations.
  • Call Graphs: LLMs frequently predicted cycles where none existed and generated extra direct calls. They struggled with indirect calls.
  • Data Flow: High rates of extra and missing edges.
  • ASTs: Tag mismatches (e.g., using expr instead of decl) were common, though in-context learning reduced these.
• Conclusion on Errors: LLMs tend to memorize the form (syntax/tags) of the output rather than understanding the underlying semantics (logic/relationships).

4. Key Contributions

1. Empirical Evidence Against "Reasoning": The paper provides strong evidence that Code LLMs do not perform static analysis in the way human programmers do. They lack the ability to build comprehensive mental models of program behavior (call graphs, data flow) through training or prompting.
2. Decoupling of Skills: The study demonstrates that the ability to generate code and the ability to perform static analysis are decoupled. Improving one does not inherently improve the other.
3. Form vs. Semantics: The authors conclude that LLMs primarily learn the "form" of programming languages (syntax, structure) rather than the "semantics" (logic, execution flow). They can mimic the output of a static analysis tool without understanding the analysis itself.
4. Benchmarking Framework: The paper establishes a rigorous framework for evaluating LLMs on static analysis tasks, including specific error taxonomies for Call Graphs, Data Flow, and ASTs.

5. Significance and Implications

• For Software Engineering: Relying on LLMs for tasks requiring deep program comprehension (e.g., security vulnerability detection, complex refactoring, impact analysis) may be risky, as the models lack the underlying static analysis capabilities humans use for these tasks.
• For Model Architecture: Current Transformer-based architectures, even with massive scale, may be insufficient for true semantic reasoning in code. Future research needs to explore hybrid approaches (e.g., integrating symbolic static analysis tools with LLMs) or new architectures that explicitly model program semantics.
• For Training Strategies: Simply pre-training on more code or fine-tuning on static analysis datasets is insufficient to teach LLMs "reasoning." The paper suggests that current training paradigms result in pattern matching rather than genuine understanding.

Conclusion: The paper concludes that Code LLMs do not do static analysis. They exhibit an "alien thought process" that relies on statistical pattern matching of code structure rather than the logical, step-by-step static analysis used by human developers.