Technical Summary: CodeSense

Problem Statement

Understanding and reasoning about code semantics—the capacity to predict software behavior beyond syntactic pattern matching—is a prerequisite for solving complex software engineering (SE) tasks such as test input generation, vulnerability detection, and bug repair. While numerous benchmarks exist for evaluating code-related Large Language Models (LLMs), they suffer from significant limitations:

1. Synthetic Data: Many benchmarks (e.g., HumanEval+, CruxEval) rely on synthetic datasets or isolated competitive programming problems, failing to capture the complexity and structure of real-world software.
2. Coarse-Grained Focus: Existing evaluations often target function-level input/output prediction or task-specific outputs (e.g., patch generation), neglecting the fine-grained semantic reasoning required for internal program behaviors, data dependencies, and control structures.
3. Lack of Ground Truth: There is a scarcity of ground-truth datasets for fine-grained semantic properties (e.g., variable values at specific execution points, loop iteration counts, pointer aliasing) derived from real-world multi-lingual projects.

Consequently, it remains unclear whether current LLMs possess the deep, execution-oriented understanding ("codesense") necessary for practical SE applications.

Methodology

The authors propose CodeSense, a benchmark and dataset constructed from real-world software repositories to evaluate fine-grained code semantic reasoning.

1. Dataset Construction

The dataset comprises 744 real-world projects across three languages:

• Python: 544 projects from the PyPIbugs dataset.
• C: 100 projects from OSS-Fuzz.
• Java: 100 projects from the SF110 dataset.

To generate ground truth, the authors developed a custom execution tracing framework and toolchain:

• Python: Used pytest for test execution and Pysnooper for tracing.
• C: Utilized the OSS-Fuzz infrastructure with GDB (GNU Debugger) for tracing.
• Java: Employed EvoSuite for test generation and a custom tool based on the Java Debugger.

This framework builds projects, runs tests, and logs execution traces containing values, data types, function calls, and memory addresses at the statement level. From these traces, the authors curated 4,483 unique samples derived from 2,125 Python, 876 C, and 875 Java functions.

2. Task Definition

CodeSense defines a spectrum of fine-grained reasoning tasks at statement, code-block, and function levels:

• Task 1 (Block-Level Semantics): Predicting execution outputs given inputs (forward reasoning) or inputs given outputs (reverse reasoning) for code blocks of varying sizes (1–3 statements) and entire functions.
• Task 2 (Statement-Level Semantics): Predicting outputs for specific statement types: arithmetic, boolean expressions, API/function calls, variable assignments, and constant assignments.
• Task 3 (Code Properties):
  • Loop Properties: Predicting iteration counts, values within loops, and values after loop execution.
  • Pointer Properties: Determining if two pointers alias (point to the same memory address) at a specific program point.
  • Branch Properties: Predicting the outcome of conditional branches.
• Task 4 (Approximation): Predicting abstract values (e.g., ranges) rather than concrete values to assess approximate reasoning capabilities.

3. Evaluation Protocol

The benchmark evaluated 14 state-of-the-art LLMs (including reasoning and non-reasoning models from families like GPT, Claude, Gemini, Llama, Qwen, and DeepSeek) using five natural language prompt templates. Evaluation metrics focused on accuracy (exact matching of generated responses against ground truth).

Key Contributions

1. CodeSense Benchmark: The first benchmark utilizing real-world Python, C, and Java projects to evaluate a spectrum of fine-grained code semantic reasoning tasks.
2. Execution Tracing Framework: An open-source, scalable toolchain that automatically generates ground-truth execution traces and semantic annotations, enabling continuous benchmark expansion while mitigating data leakage.
3. Comprehensive Evaluation: An empirical analysis of 14 SOTA LLMs across six research questions (RQs) covering code size, statement types, critical SE properties, prompting strategies, approximation, and language differences.
4. Public Resources: Release of the dataset, code, and a public leaderboard to support reproducibility and future research.

Results

The evaluation reveals a significant performance gap between current LLMs and the requirements for fine-grained code reasoning:

• General Struggle: Models struggle even with single-statement reasoning. In C datasets, top models (e.g., Claude 3.5, GPT-4o-mini) achieved under 30% accuracy on single-statement blocks. Python results were slightly better but no model exceeded 50%.
• Reverse Semantics: LLMs perform particularly poorly on "reverse" semantic inference (predicting inputs from outputs), with even the best models achieving only ~12% accuracy for small functions.
• Statement Types: Arithmetic expressions and API calls were the most difficult. Models lacked knowledge of the execution semantics of common third-party libraries (e.g., os, sys, math). They performed better on boolean comparisons and simple assignments.
• Code Properties:
  • Loops: Predicting loop iteration counts was the easiest task, likely due to pattern recognition (e.g., range(100)), but predicting values after loop execution remained difficult.
  • Pointers & Branches: Models performed better on binary classification tasks (pointer aliasing, branch outcomes) than on value prediction, often leveraging simple pattern correlations (e.g., p=q implies aliasing).
• Prompting Strategies:
  • Chain-of-Thought (CoT) provided limited benefit for statement prediction but helped with complex loop reasoning.
  • In-context learning (few-shot) improved performance, especially when examples were highly relevant to the query (RAG style).
  • Models could predict abstract values better than concrete values only when provided with few-shot examples of the abstraction; zero-shot abstraction failed.
• Language Differences: Java and Python were generally easier for models to reason about than C, likely due to higher-level abstractions and greater representation in training data.
• Model Performance: Among the 14 models, Claude 3.5 consistently achieved the best performance, though it still faced significant challenges with fine-grained tasks.

Significance and Claims

The paper claims that CodeSense establishes a new standard for evaluating the "codesense" of LLMs in practical software engineering contexts. The authors argue that while prompting techniques (CoT, in-context learning) offer marginal improvements, the fundamental lack of deep code semantics in current models limits their ability to solve real-world SE tasks.

The work highlights that:

1. Current models fail to generalize semantic reasoning from code text alone without execution traces or specific patterns.
2. The proposed framework and dataset provide a necessary foundation for future benchmark construction and model post-training.
3. The inability to reason about fine-grained semantics (arithmetic, API calls, control flow) remains a critical bottleneck for deploying LLMs in safety-critical or complex engineering scenarios.

The authors position CodeSense not as a solution to these limitations, but as a diagnostic tool to expose them, offering a "strong basis" for future research into improving model capabilities in semantic reasoning.