Exploring the Potential of Program Flowcharts on Code Generation Using Multimodal LLMs

Technical Summary: Exploring the Potential of Program Flowcharts on Code Generation Using Multimodal LLMs

Problem Statement

While Large Language Models (LLMs) have advanced significantly in automatic code generation, most existing approaches rely solely on textual prompts. In software development, visual artifacts like flowcharts are widely used to aid comprehension and design, yet their specific impact on the performance of multimodal LLMs remains underexplored. Furthermore, there is a lack of understanding regarding the optimal level of detail required in these visual prompts (e.g., whether highly abstracted diagrams are sufficient or if detailed flowcharts are necessary) and the cost-effectiveness of integrating such visual data into the generation process.

Methodology

The study conducted a systematic evaluation using GPT-4o, a state-of-the-art multimodal LLM, to assess code generation performance on competitive programming tasks.

• Dataset: The researchers utilized 289 coding problems from the AtCoder platform (ranging from difficulty levels A to D), paired with official solution code (Python) and test cases.
• Flowchart Generation: Since AtCoder does not provide flowcharts, the authors used Visustin, an automated tool, to generate flowcharts from the solution code.
• Experimental Design: The study compared nine prompting strategies across four research questions (RQs):
  1. RQ1 (Baseline vs. Visual): Comparing Problem Statements (PS) alone versus PS combined with full flowcharts (FC).
  2. RQ2 (Abstraction Levels): Testing PS + FC with varying levels of abstraction (Depth 0, 1, and 2) based on code indentation depth. Depth 0 represents global variables only; Depth 1 includes top-level control structures; Depth 2 includes nested structures.
  3. RQ3 (Few-Shot Learning): Evaluating the impact of providing 1-Shot and 2-Shot examples (containing both problem statements and flowcharts) alongside the target prompt.
  4. RQ4 (Cost Analysis): Analyzing token usage and API costs for each strategy.
• Evaluation: Code generation was executed five times per prompt to account for LLM randomness. Accuracy was measured as the percentage of generated solutions passing all test cases.

Key Results

• Impact of Flowcharts (RQ1): Providing flowcharts alongside problem statements improved code generation accuracy by 4–10% on average. The improvement was most pronounced for difficult problems (Levels C and D), where accuracy increased by up to 13%. The average accuracy rose from 74.93% (PS only) to 85.11% (PS + FC).
• Optimal Abstraction (RQ2): There is a positive correlation between flowchart detail and performance.
  • Depth 0 (highly abstracted) performed worse than the PS-only baseline (71.93% vs. 74.93%).
  • Depth 2 (more detailed) achieved an average accuracy of 83.13%, significantly outperforming the baseline.
  • The marginal gain between Depth 2 and full flowcharts was small (~2%), suggesting that capturing control structures up to a certain depth is sufficient for high performance.
• Few-Shot Learning (RQ3):
  • 1-Shot learning provided sustainable improvements, increasing accuracy by approximately 6% over Zero-Shot with abstracted flowcharts.
  • 2-Shot learning yielded only marginal gains or, in some cases, a slight decrease in accuracy compared to 1-Shot, suggesting diminishing returns with additional examples.
• Cost Efficiency (RQ4): While adding flowcharts increased input token counts (and thus input costs) by roughly 1.45x to 2.34x, the total cost per generation remained low (approx. $0.0012 to $0.0045). Notably, the presence of flowcharts often reduced output token counts (by up to ~34% for full flowcharts), as the model generated more concise and precise code. This resulted in a net cost-effective improvement.

Significance and Claims

The paper claims that integrating program flowcharts as visual prompts is a cost-effective and high-impact strategy for enhancing multimodal LLMs in code generation. Key implications include:

1. Visual Guidance: Flowcharts act as effective comprehension aids, particularly for complex algorithmic problems where textual descriptions alone may be insufficient.
2. Detail vs. Effort Trade-off: While highly detailed flowcharts yield the best results, abstracted flowcharts (Depth 2) offer a strong balance between manual effort (or generation effort) and performance gains.
3. Practical Workflow: The study suggests that in practical development, providing a problem statement with a flowchart detailing the overall structure (Depth 1 or 2) and a single relevant example (1-Shot) offers the best trade-off between accuracy and effort.
4. Economic Viability: The approach is economically viable; the slight increase in input costs is offset by reduced output costs and significantly higher accuracy, making it a viable addition to LLM-driven development workflows.

The authors conclude that visual design elements are not merely supplementary but are critical for guiding multimodal LLMs in complex programming tasks, opening avenues for future research into other diagram types (e.g., UML) and fine-tuning models specifically for diagram-driven development.