COBOLAssist: Analyzing and Fixing Compilation Errors for LLM-Powered COBOL Code Generation

1. Problem Statement

Legacy systems relying on COBOL (Common Business-Oriented Language) remain critical for global infrastructure (banking, insurance, government), yet they face a severe maintenance crisis due to a shrinking pool of skilled developers. While Large Language Models (LLMs) offer a potential solution for code generation, they struggle significantly with COBOL due to:

• Low-Resource Nature: Limited open-source COBOL repositories and training data.
• Rigid Syntax: COBOL's verbose, structure-heavy syntax (divisions, sections, paragraphs) and strict typing rules are difficult for models trained on modern languages to master.
• High Failure Rates: Existing LLMs generate COBOL code with low compilation success rates and frequent semantic errors.
• Lack of Repair Mechanisms: Prior research focuses on general code generation or error fixing in modern languages (Python, Rust), with little attention paid to automated compilation error repair specifically for COBOL.

2. Methodology

The study employs a three-pronged methodology: Error Categorization, Framework Design (COBOLAssist), and Empirical Evaluation.

A. Compilation Error Categorization

The authors conducted the first systematic analysis of compilation errors in LLM-generated COBOL code, comparing them against historical data from human-written code (Litecky, 1974). They categorized 980 compilation errors from 876 generated programs into three main groups:

1. Incomplete Code Errors:
   • Incomplete Block Termination: Missing keywords like END-IF, END-PERFORM, or END PROGRAM.
   • Unterminated Statements: Missing periods (.) or delimiters at the end of statements.
2. Syntax Errors:
   • Incorrect Use of Program Structures: Structural violations, such as duplicating LINKAGE SECTION or incorrect nesting.
   • Incorrect Use of Reserved Words: Using keywords as variable names.
   • Incorrect Use of Built-in Functions: Misapplying functions (e.g., using MOD as an operator instead of FUNCTION MOD).
   • Undefined Objects: Referencing variables before declaration.
   • Incorrect Use of Variables: Scope violations or type mismatches.
3. Type-Related Errors:
   • Incorrect Data Type: Mismatches in PIC clauses (e.g., assigning alphanumeric to numeric fields).

Key Finding: LLMs exhibit unique error patterns compared to humans, specifically a high rate of Incorrect Use of Program Structures (35.1% vs. 19.8% in humans) and unique errors like Incorrect Use of Built-in Functions (17.2%) and Incomplete Block Termination (5.6%).

B. COBOLAssist Framework

The authors propose COBOLAssist, a compiler-guided self-debugging framework.

• Workflow:
  1. Generation: An LLM generates initial COBOL code based on a natural language prompt.
  2. Compilation: The code is compiled using the GnuCOBOL compiler.
  3. Feedback Loop: If compilation fails, the error log is fed back into the LLM with a specific prompt instructing it to act as an experienced COBOL engineer to fix the errors.
  4. Iteration: This cycle repeats (up to a maximum of 3 iterations in the main study) until the code compiles successfully or the iteration limit is reached.
• Prompt Engineering: The system uses a structured prompt separating the "Original Code" and "Compiler Error Log" to guide the model's reasoning.

C. Experimental Setup

• Benchmark: COBOLEval (146 programming tasks translated from HumanEval-Python).
• Models Evaluated:
  • General-purpose: GPT-3.5-Turbo, GPT-4, GPT-4o-mini, GPT-4o.
  • COBOL-specific: mAInframer (7B, 13B, 34B), fine-tuned on CodeLlama.
• Metrics:
  • Compilation Success Rate (CSR): Percentage of code that compiles without errors.
  • Pass@1: Percentage of tasks where the top-1 generated solution passes all unit tests (functional correctness).

3. Key Results

RQ1: Error Patterns

• LLMs struggle most with structural logic (35.1% error rate) compared to humans (19.8%).
• Two error types are unique to LLMs: Incorrect Use of Built-in Functions and Incomplete Block Termination, highlighting a gap in understanding COBOL's specific semantics.
• Human errors are more diverse ("Others" category), while LLM errors are concentrated in specific structural and semantic categories.

RQ2: Effectiveness of COBOLAssist

COBOLAssist significantly outperforms both baseline generation and zero-shot self-refinement (without compiler feedback).

• Compilation Success Rate (CSR) Improvements:
  • GPT-4o: Increased from 41.8% to 95.89%.
  • GPT-4o-mini: Increased from 29.5% to 64.38%.
  • mAInframer-34B: Increased from 74.66% to 97.94% (when paired with GPT-4o for debugging).
• Functional Correctness (Pass@1) Improvements:
  • GPT-4: Improved from 9.1 to 22.6.
  • GPT-4o: Improved from 16.4 to 29.45.
• Efficiency: The framework reduces compilation errors by an average of 44% to 85% across models, typically requiring fewer than 2 iterations on average.

RQ3: Unresolved Errors

Despite high success rates, some errors persist:

• Remaining Challenges: The most difficult errors to fix automatically are Incorrect Use of Program Structures, Incorrect Use of Built-in Functions, and Incomplete Block Termination.
• Reason: These require deep contextual understanding and multi-line reasoning (e.g., tracking nested IF blocks across long code segments), which current LLMs struggle with even with feedback.
• Semantic Gap: While mAInframer-34B achieved the highest CSR (97.94%), its Pass@1 remained low (11.67), indicating that syntactic correctness does not guarantee functional correctness.

4. Key Contributions

1. First Empirical Study: The first systematic categorization of compilation errors specifically in LLM-generated COBOL code, contrasting them with human-written errors.
2. COBOLAssist Framework: A novel, compiler-guided self-debugging technique that iteratively leverages compiler feedback to fix syntax and structural errors.
3. Performance Validation: Demonstrated that COBOLAssist can boost compilation success rates from ~30-40% to >90% for state-of-the-art models.
4. Insight into Limitations: Identified that while LLMs can fix simple syntax errors, they still fail on complex structural semantics, suggesting a need for hybrid AI-human approaches or syntax-aware tools (e.g., AST-based analysis).

5. Significance and Implications

• Legacy Modernization: Provides a practical path forward for automating the maintenance and modernization of critical COBOL systems, addressing the developer shortage.
• Developer Guidance: Alerts developers using LLMs for COBOL to expect specific structural and semantic errors (e.g., missing terminators) that differ from human coding mistakes.
• Research Direction: Highlights the need for future research into syntax-aware repair (incorporating Abstract Syntax Trees) and semantic-aware debugging to bridge the gap between compilation success and functional correctness.
• Generalizability: While focused on COBOL, the compiler-feedback loop methodology offers a blueprint for improving code generation in other rigid, legacy, or low-resource languages.