Less Is More: Measuring How LLM Involvement affects Chatbot Accuracy in Static Analysis

1. Problem Statement

The paper addresses the challenge of bridging the gap between unstructured natural language (developer queries, requirements) and structured formal languages (specifically CPGQL, the query language for Joern static analysis tools).

While Large Language Models (LLMs) are increasingly used to translate natural language into code or queries, their reliability in formally structured domains is uncertain. Existing systems vary in how much they delegate to the LLM, but few treat the degree of LLM involvement as an independent variable. The core problem is determining the optimal balance: how much should the LLM generate directly versus how much should it be constrained by deterministic code or intermediate representations to ensure semantic correctness in static analysis?

2. Methodology

The authors conducted a controlled empirical study comparing three distinct architectures along a spectrum of LLM autonomy. The evaluation was performed on a custom benchmark of 20 code analysis tasks across three complexity tiers (Structural, Data Flow, and Composite) using real-world Java projects (Apache Commons Lang and OWASP WebGoat).

The Three Architectures

1. A1: Direct Generation

   • Mechanism: The LLM receives the natural language request and context (documentation/examples) and directly generates a CPGQL string.
   • Constraint: None. The output space is unbounded; the LLM must handle Scala-flavored DSL syntax, method chaining, and graph traversal semantics.
   • Retry: Up to 3 attempts on execution errors.

2. A2: Structured Intermediate (The "Less Is More" Approach)

   • Mechanism: The LLM generates a JSON object conforming to a fixed, well-typed schema (defining query type, scope, filters, etc.). A deterministic mapper then translates this JSON into the correct CPGQL invocation.
   • Constraint: The LLM never sees CPGQL syntax. Its task is reduced to classification and field extraction within a constrained schema.
   • Retry: Up to 3 attempts on JSON parsing/schema validation errors.

3. A3: Tool-Augmented Agentic Generation

   • Mechanism: The LLM operates in a ReAct loop, selecting from a set of 5 tools (e.g., find_methods, trace_data_flow) and providing arguments in a multi-step process to synthesize a final answer.
   • Constraint: The LLM controls the workflow but must make correct tool selections and argument choices at every step.
   • Retry: Up to 10 steps per task.

Experimental Setup

• Models: Four open-weight models in a 2×2 design:
  • Families: Llama (3.1/3.3) and Qwen (2.5).
  • Scales: Small (7–8B parameters) and Large (70–72B parameters).
• Trials: 720 total trials (20 tasks × 3 repetitions × 4 models × 3 approaches, with some exclusions due to infrastructure limits).
• Metrics: Result match rate (semantic correctness against ground truth), exact string match, token consumption, and execution success rate.

3. Key Contributions

1. Benchmark Creation: A publicly available benchmark of 20 natural-language-to-CPGQL translation tasks across three complexity tiers, machine-validated against Joern 4.0.
2. Architectural Comparison: A controlled study isolating the variable of "LLM involvement," comparing direct generation, structured intermediates, and agentic tool use.
3. Empirical Evidence for Constrained Generation: Demonstrating that constraining LLM output to a structured intermediate representation significantly outperforms both unconstrained generation and complex agentic loops in formal domains.
4. Model-Size Interaction Analysis: Revealing that the benefits of structured intermediates are highly dependent on model scale, with small models facing a bottleneck in schema compliance.

4. Key Results

A. Structured Intermediates (A2) Outperform All Others

• Performance: A2 achieved the highest result match rates across all models.
  • Large Models: Outperformed Direct Generation (A1) by 15–25 percentage points (e.g., Qwen 72B: 58.3% vs. 43.3%).
  • Small Models: Showed modest gains (3–5 pp) but were limited by schema compliance failures.
• Efficiency: A2 consumed significantly fewer tokens than A3 while achieving higher accuracy.

B. Agentic Generation (A3) is Inefficient and Inaccurate

• Performance: A3 performed the worst across all models (e.g., Qwen 72B: 25.0% match rate).
• Cost: A3 consumed approximately 8× more tokens on average than A2 and took significantly longer (median 6,756 tokens vs. <2,000 for A1/A2).
• Failure Mode: Errors compounded across steps. Even with high per-step accuracy, multi-step chains failed to converge on correct answers.
• No Unique Coverage: The set of tasks solved by A3 was a strict subset of those solved by A2. A3 offered no complementary value to justify its cost.

C. Model Size Interaction

• Large Models: Can reliably fill the JSON schema (100% execution success on A2), reaping the full architectural benefits.
• Small Models: Struggle with schema compliance (35–47% failure rate in JSON parsing/validation). For them, the bottleneck shifts from query logic to formatting constraints, limiting the advantage of A2.

D. Token Cost vs. Accuracy

• There is no correlation between compute cost and accuracy. A3's high token consumption did not translate to better results; in fact, it correlated with lower accuracy due to error compounding.

5. Significance and Implications

• "Less is More" Principle: In formally structured domains (like static analysis), constraining the LLM's output space yields better results than giving it full autonomy. The optimal strategy is to let the LLM handle natural language understanding and classification into a schema, while delegating the actual query construction to deterministic code.
• Limitations of Agentic Loops: For tasks requiring precise, deterministic outcomes, iterative tool use (ReAct) introduces unnecessary complexity and error compounding. It is not a panacea for improving accuracy in DSL generation.
• Evaluation Methodology: The paper highlights that exact string matching underestimates LLM correctness. Semantic evaluation (executing queries and comparing results) is necessary for accurate assessment in DSL generation.
• Future Directions: For small models, techniques like guided decoding or constrained generation could enforce schema compliance at the token level, potentially unlocking the benefits of structured intermediates for smaller parameter counts.

In conclusion, the paper argues that for static analysis and similar formal domains, the most effective architecture minimizes the LLM's role to intent extraction and schema filling, relying on deterministic code to handle the syntactic and semantic rigor of the target language.