💻 computer science

COBOL-Coder: Domain-Adapted Large Language Models for COBOL Code Generation and Translation

This paper introduces COBOL-Coder, a domain-adapted large language model developed through an automated data curation pipeline that significantly outperforms general-purpose LLMs in COBOL code generation and Java-COBOL translation tasks, as validated by both benchmark metrics and expert developer surveys.

Original authors: Anh T. V. Dau, Shin Hwei Tan, Jinqiu Yang, Nghi D. Q. Bui, Anh Tuan Nguyen

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anh T. V. Dau, Shin Hwei Tan, Jinqiu Yang, Nghi D. Q. Bui, Anh Tuan Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of software as a massive, bustling city. Most of the new buildings are modern skyscrapers made of glass and steel, built with the latest tools (languages like Python and Java). But deep in the city's foundation, there are still massive, ancient stone fortresses running the banks, insurance companies, and government systems. These fortresses are built with COBOL, a programming language invented in the 1960s.

The problem? The city's newest, smartest construction robots (Large Language Models or LLMs) are amazing at building glass skyscrapers. But when you ask them to fix or build a stone fortress, they get confused. They don't know the old rules, the specific stone-cutting techniques, or how the ancient blueprints work. They try to build a glass window into a stone wall, and it falls apart.

This paper introduces COBOL-Coder, a specialized robot designed specifically to understand and build these ancient stone fortresses. Here is how they did it, explained simply:

1. The Problem: The "Language Barrier"

Think of COBOL as a very old, strict dialect. It's not just about the words; it's about the grammar, the formatting, and the specific way the "stone" (data) must be laid.

The Old Robots: General AI models (like GPT-4) are like polyglots who speak 100 modern languages fluently but have never seen a stone fortress. They try to guess, but their guesses are usually wrong. They can't even get the basic structure right.
The Data Scarcity: There are very few public blueprints (code) for these fortresses because they are kept secret in bank vaults. The robots didn't have enough practice material to learn.

2. The Solution: Building a Specialized Training School

The researchers didn't just give the robot more books; they built a specialized training academy. They created COBOL-Coder through three clever steps:

Step A: The "Self-Healing" Library (Data Curation)
They dug up old blueprints from public archives (GitHub). But many were broken or incomplete.
- The Analogy: Imagine a robot librarian who finds a torn, muddy blueprint. Instead of throwing it away, the librarian uses a "magic compiler" (a tool that checks for errors) to see what's wrong. Then, they ask a super-smart AI (GPT-4) to act as a master mason and fix the blueprint. They repeat this until the blueprint is perfect.
- Result: They turned thousands of broken, messy blueprints into a clean, perfect library of 31,000+ working COBOL programs.
Step B: The "Translator" Gym (Synthetic Data)
Since there weren't enough COBOL blueprints, they used a trick. They took modern, easy-to-read blueprints (Java code) and asked the AI to translate them into the ancient COBOL dialect.
- The Analogy: It's like taking a modern English novel and asking a translator to rewrite it in 1960s business English. They then checked the translation twice: once to see if it made sense, and again by translating it back to English to see if the story stayed the same. This created a massive new set of practice problems.
Step C: The "Mentor" (Instruction Tuning)
They didn't just feed the robot raw blueprints. They created a "Question and Answer" textbook. They took the blueprints and wrote questions like, "How do you calculate interest?" and provided the correct COBOL answer. This taught the robot how to think like a COBOL developer, not just memorize code.

3. The Results: The Specialized Robot Wins

They put COBOL-Coder to the test against the general-purpose giants (like GPT-4o and other open-source models).

The Test: They asked the robots to build small COBOL programs and translate Java code into COBOL.
The Outcome:
- General Robots: Most failed completely. They produced code that wouldn't even compile (like a car that wouldn't start). GPT-4o managed to get about 42% of the code working.
- COBOL-Coder: It got 74% of the code working perfectly. When translating modern code back to the ancient language, it was the only one that could do it with any success, while the others scored near zero.

4. The Human Verdict: The "Senior Mason" Review

Numbers are great, but do the actual builders like it? The researchers asked three experienced COBOL developers (the "Senior Masons") to review the code.

The Feedback: The developers said COBOL-Coder felt "pragmatic" and "production-ready." It understood the old-school structure and didn't try to be fancy.
The General Robots: The developers felt the general robots were either "inconsistent" or "over-engineered" (trying to use modern tricks that didn't fit the old system).
The Takeaway: The developers saw COBOL-Coder as a helpful junior apprentice who could draft the work quickly, saving them time. They didn't see the general robots as useful at all for this specific job.

The Big Lesson

The paper teaches us that size isn't everything. A massive, general-purpose robot isn't necessarily better than a smaller, specialized one if the task requires deep, niche knowledge.

To fix the world's ancient systems, we don't just need bigger brains; we need specialized training. We need to teach our AI the specific dialect, the strict rules, and the history of the language, rather than just hoping it can guess its way through.

In short: COBOL-Coder is the result of taking a smart AI, giving it a massive, high-quality textbook of COBOL, and training it until it speaks the language of the mainframe fluently, while the other AIs are still trying to learn the alphabet.

1. Problem Statement

Despite COBOL's critical role in banking, insurance, and government mainframe systems, existing Large Language Models (LLMs) perform poorly on COBOL-related tasks. The paper identifies three primary challenges:

Data Scarcity: COBOL is a low-resource language; most production code is proprietary and not publicly available, leading to insufficient training data for general-purpose LLMs.
Benchmark Limitations: Existing benchmarks (e.g., COBOLEval) focus primarily on code generation and lack support for code translation or modernization scenarios.
Structural Differences: COBOL differs significantly from modern languages in syntax, formatting, and data layouts, causing general-purpose models to fail at generating compilable or idiomatic code.

Current state-of-the-art models (e.g., GPT-4o, CodeLlama) achieve near-zero compilation success rates (CSR) and functional correctness (Pass@1) on COBOL tasks, highlighting a fundamental gap in handling legacy programming languages.

2. Methodology

The authors propose a comprehensive framework involving data curation, model adaptation, and benchmark construction.

A. Automated Data Curation Pipeline

To overcome data scarcity, the authors constructed a high-quality COBOL training corpus using three sources and a multi-stage filtering process (Figure 1 in the paper):

Source 1: Real-world COBOL (GitHub):
- Collected 40,829 unique files from GitHub.
- Stage 1 (Compiler-Guided Repair): Used GnuCOBOL to compile files. Uncompilable files were sent to an LLM (GPT-4o) with compiler error logs to iteratively repair syntax errors until the code compiled successfully. This yielded 31,492 compilable programs.
Source 2: Synthetic COBOL (Cross-Language Translation):
- Translated high-resource Java code (from The Stack v2) into COBOL using an LLM.
- Stage 1: Applied the same compiler-guided repair pipeline to ensure syntactic validity.
- Stage 2 (Similarity-Based Validation): Ensured semantic consistency via two checks:
  - LLM Scoring: An LLM scored the functional similarity between the original Java and translated COBOL.
  - Back-Translation: The COBOL was translated back to Java, and an AST-based similarity score (using CodeBERTScore) compared the back-translated Java to the original.
- This process yielded ~173,000 high-quality Java-COBOL pairs.
Source 3: Knowledge Sources:
- Curated textbooks and technical documentation, converted into Question-Answer pairs to teach COBOL concepts and mainframe conventions.

B. Model Architecture and Training

Base Model: The authors fine-tuned the Qwen2.5-Coder family (7B and 14B parameters), chosen for its strong instruction-following capabilities.
Fine-Tuning: The models were instruction-tuned on the curated dataset (approx. 600M+ tokens total) using standard hyperparameters (AdamW optimizer, cosine learning rate schedule).
Resulting Model: COBOL-Coder, a domain-adapted LLM specialized for COBOL syntax, structure, and mainframe practices.

C. Benchmark Construction

COBOL-JavaTrans: The first benchmark for bidirectional COBOL–Java translation. It adapts 143 tasks from HumanEval and HumanEval-X, providing reference implementations in both languages to evaluate modernization (COBOL $\to$ Java) and legacy migration (Java $\to$ COBOL).

3. Key Contributions

Automated Data Pipeline: A novel pipeline combining compiler-guided self-debugging and multi-stage similarity filtering to construct a large-scale, high-quality COBOL instruction-tuning corpus.
COBOL-Coder: A specialized LLM (7B and 14B variants) that significantly outperforms general-purpose and other code-specialized LLMs on COBOL tasks.
COBOL-JavaTrans Benchmark: A new evaluation suite for bidirectional translation, addressing the lack of benchmarks for legacy system modernization.
Practitioner Evaluation: A user study with experienced COBOL developers providing qualitative insights into the usability and "production-readiness" of LLM-generated code.

4. Experimental Results

Code Generation (COBOLEval & COBOLCodeBench)

Compilation Success Rate (CSR): COBOL-Coder-14B achieved 73.95%, compared to 41.8% for GPT-4o. Most open-source baselines (CodeGemma, CodeLlama, StarCoder2) achieved 0%.
Functional Correctness (Pass@1): COBOL-Coder-14B achieved 49.33, significantly outperforming GPT-4o (16.4) and all other baselines (mostly <1.5 or 0).
Observation: General-purpose LLMs failed to generate compilable programs on the harder COBOLCodeBench, while COBOL-Coder achieved non-trivial results (26.09% CSR).

Code Translation (COBOL-JavaTrans)

COBOL-to-Java: COBOL-Coder-14B achieved 97.90% CSR and 83.91 Pass@1, approaching the performance of much larger models like GPT-oss-120B.
Java-to-COBOL: This task is significantly harder. COBOL-Coder-14B achieved 72.03% CSR and 34.93 Pass@1. In contrast, general-purpose LLMs (GPT-4, GPT-4o) scored near zero (<5% CSR).

User Study (Practitioner Feedback)

Rankings: Experienced developers ranked COBOL-Coder first in all Java-to-COBOL translation tasks and first/tied-first in most generation tasks.
Qualitative Insights: Developers perceived COBOL-Coder as more "production-oriented," with better adherence to enterprise coding patterns, file handling, and batch processing logic. General-purpose models were criticized for structural errors and "over-engineering."

5. Significance and Implications

Data Quality > Model Scale: The study demonstrates that for low-resource legacy languages, targeted domain adaptation and high-quality data curation are more effective than simply scaling up model parameters.
Asymmetry in Translation: The paper highlights that translating to a legacy language (Java $\to$ COBOL) is fundamentally harder than translating from it, requiring specialized modeling strategies.
Human-in-the-Loop: The results suggest LLMs are best utilized as "junior developer" draft generators to reduce cognitive load, rather than fully autonomous agents, especially in complex mainframe environments involving copybooks and JCL.
Legacy Modernization: By providing a reliable tool for bidirectional translation, this work offers a practical pathway for modernizing critical legacy systems without losing functional correctness.

In conclusion, COBOL-Coder sets a new state-of-the-art for legacy language processing, proving that domain-specific adaptation can bridge the gap between modern AI capabilities and the rigid requirements of enterprise mainframe systems.