💻 computer science

Deterministic vs. LLM-Controlled Orchestration for COBOL-to-Python Modernization

This paper presents a controlled empirical study demonstrating that deterministic orchestration outperforms LLM-controlled agentic workflows in COBOL-to-Python modernization by achieving comparable translation accuracy while significantly improving robustness, reducing performance variability, and lowering operational costs by up to 3.5x.

Original authors: Naing Oo Lwin, Rajesh Kumar

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Naing Oo Lwin, Rajesh Kumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

For decades, the financial and government sectors have relied on massive, intricate computer programs written in a language called COBOL. These systems run the world's banking transactions and social security payments, yet the experts who built them are retiring, and the code itself is often decades old, poorly documented, and difficult to change. To keep these vital services running, organizations must translate this ancient code into modern languages like Python, a process that requires extreme precision because even a tiny error can cause a system to fail. Recently, a new type of artificial intelligence known as a large language model has emerged as a potential tool for this task. These models can read code and write new code, but they often work by acting like autonomous agents: they decide for themselves which steps to take, when to retry a task, and how to fix mistakes as they go. This raises a critical question for engineers: is it better to let the artificial intelligence drive the entire process, deciding every move in real time, or is it more effective to keep the process on a strict, pre-defined track where the computer follows a fixed set of rules?

A team of researchers at Bucknell University and Astrio set out to answer this question by conducting a controlled experiment to see how these two different approaches handle the difficult job of converting COBOL code into Python. They built a system called ATLAS to act as the translator, but they designed the experiment so that the only thing that changed between tests was who was in the driver's seat. In one version, the artificial intelligence was given full control, allowed to choose its own path, select its own tools, and decide when to stop or start over. In the other version, the same artificial intelligence was used to write the code, but a rigid, unchanging set of rules dictated exactly what steps to take, in what order, and how many times to try again if something went wrong. By keeping the brain of the system—the language model—exactly the same in both scenarios, the researchers could isolate the effect of the control method itself, stripping away all other variables to see which strategy truly worked better.

The results of this study revealed a clear trade-off between flexibility and reliability. When the artificial intelligence was allowed to control the process, it frequently managed to produce a working program, often succeeding in generating a result that could run without crashing. However, this success came with a hidden cost: the results were inconsistent. Because the model was making its own decisions about how to proceed, the same starting code could lead to very different outcomes on different tries, sometimes producing a perfect translation and other times failing in unpredictable ways. Furthermore, this "agentic" approach was incredibly expensive. The model would often wander down long, winding paths of reasoning, asking itself questions and trying multiple strategies that were not necessary, consuming vast amounts of computational resources. In some cases, the flexible approach used more than three times as many tokens—the basic units of information the model processes—as the fixed approach to achieve a similar result.

In contrast, the system that followed a fixed, deterministic path produced results that were far more stable and predictable. While the overall ability to translate the code was just as good as the flexible version, the rigid system rarely failed in the worst cases. It did not get lost in loops of unnecessary thinking, and it did not vary its behavior from one run to the next. Most importantly, it was significantly cheaper to run. By sticking to a strict schedule of steps and only using the artificial intelligence to write the code rather than to plan the journey, the system reduced the cost of translation by a factor of three to three and a half. The researchers found that for tasks like modernizing legacy software, where the steps are well-defined and the results can be checked against strict rules, letting the artificial intelligence drive the car was not only more expensive but also less reliable than having a human-designed map guide the way.

This study suggests that the future of using artificial intelligence in complex engineering tasks may not lie in giving the machines total freedom to decide how to work. Instead, the most effective approach appears to be embedding these powerful models within a structured framework where the rules of engagement are fixed and the path is clear. The artificial intelligence remains the expert writer, capable of understanding complex instructions and generating new code, but the orchestration of the work—the planning, the timing, and the safety checks—remains under strict, deterministic control. This ensures that the process is not only capable of producing high-quality results but also remains robust, predictable, and economically viable for the large-scale, critical systems that keep our modern world running.

Technical Summary: Deterministic vs. LLM-Controlled Orchestration for COBOL-to-Python Modernization

Problem Statement

Modernizing legacy COBOL systems is a critical challenge due to scarce expertise, massive codebases, and strict correctness requirements. While Large Language Models (LLMs) have shown promise in code translation, recent systems increasingly rely on agentic workflows where the LLM dynamically controls multi-step tool execution, including tool selection, ordering, and retry logic. However, it remains unclear whether delegating execution control to the LLM improves correctness, robustness, or efficiency in structured software engineering workflows (such as legacy modernization) compared to fixed, deterministic execution policies. Existing literature often conflates orchestration strategy with model capability or prompt design, making it difficult to isolate the specific impact of the execution control mechanism.

Methodology

The authors present a controlled empirical study using the ATLAS (Autonomous Transpilation for Legacy Application Systems) framework to compare two orchestration strategies: Deterministic Orchestration and LLM-Controlled Orchestration.

Experimental Design

To isolate orchestration as the sole experimental variable, the study holds the following constant across all runs:

Language Models: Identical models, versions, and decoding parameters (temperature, seeds).
Prompts: Identical system prompts, task instructions, and formatting constraints.
Tools: Identical tool sets (e.g., read_file, write_file, run_command, git) with the same interfaces and permissions.
Inputs: The same source COBOL programs and test inputs.
Configuration: Identical timeout limits and configuration flags.

Orchestration Strategies

Deterministic Orchestration: The sequence of tool invocations, validation stages, and retry behaviors is fixed and predefined. The LLM is responsible only for code generation within specific stages. Execution flow does not depend on intermediate LLM outputs; conditional branching is governed by deterministic predicates over system state (e.g., file modification status).
LLM-Controlled Orchestration: The LLM acts as a central orchestrator, dynamically selecting tools, determining execution order, choosing repair strategies upon failure, and deciding when to terminate. Execution flow emerges from the model's generation process and is influenced by stochastic sampling.

Dataset and Evaluation

Dataset: The NIST COBOL85 Test Suite, comprising hundreds of self-contained COBOL programs with executable test harnesses and well-defined expected outcomes.
Metrics:
- Computational Accuracy (CA): Functional equivalence between the generated Python program and the reference COBOL implementation over test inputs.
- Success Rate (SR): The frequency with which a translation pipeline produces a runnable, test-passing program across repeated stochastic runs ( $N$ independent executions).
- Worst-Case Robustness: Measured via the 5th percentile of CA (P5-CA) and Conditional Value at Risk (CVaR) to capture tail-risk failures.
- Efficiency: Token consumption per successful translation and estimated operational cost.

Key Contributions

Controlled Framework: The authors introduce a unified experimental framework that isolates execution control as the sole variable, enabling a direct comparison between deterministic and agentic orchestration without confounding factors.
Robustness vs. Frequency Trade-off: The study demonstrates that while both strategies achieve comparable overall translation capability, deterministic orchestration significantly improves worst-case robustness (reducing tail-risk failures and performance variability), whereas LLM-controlled orchestration yields higher success rates in producing runnable code across repeated attempts.
Efficiency Analysis: The paper provides a quantitative analysis showing that deterministic orchestration reduces token consumption by up to 3.5×, leading to substantially lower operational costs compared to adaptive agentic workflows.

Results

Correctness: Deterministic orchestration consistently achieved higher mean Computational Accuracy (CA) and stronger worst-case robustness (higher P5-CA and CVaR) across multiple models (including Claude-Sonnet-4-5, GPT-5.1-Codex-Max, and Grok-Code-Fast-1).
Success Rate: LLM-controlled orchestration achieved a higher Success Rate (SR), indicating that adaptive execution more frequently resulted in compilable, test-passing programs in individual runs. However, this came at the cost of heavier performance tails (more severe failures in the worst cases).
Cost and Efficiency: LLM-controlled orchestration required significantly more tokens (e.g., 1.75M–2.25M tokens for complex modules vs. 400k–700k for deterministic). This resulted in operational costs up to 3.5× higher for agentic workflows (e.g., >$140 vs. ~$40 per successful translation for Sequential I/O tasks).
Test Suite Consistency: Aggregate test statistics showed that both strategies could translate the same set of programs, but deterministic orchestration provided more consistent correctness guarantees in the final outputs.

Significance and Claims

The paper argues that for structured modernization workflows characterized by explicit validation stages and deterministic dependencies, fixed execution policies provide a more stable and cost-efficient alternative to fully agentic orchestration without sacrificing translation quality.

Design Implication: Execution control should be treated as a first-class systems design decision rather than an implicit consequence of LLM reasoning. In tasks with clear transformation pipelines and strong validation signals, separating execution control from generative reasoning limits error accumulation and improves predictability.
Scope of Applicability: The findings suggest that while agentic orchestration may remain valuable for open-ended, exploratory tasks (e.g., requirements discovery), structured legacy modernization benefits from deterministic defaults.
Modesty: The authors acknowledge limitations, noting that the results are specific to COBOL-to-Python modernization and the NIST COBOL85 suite, which may not fully capture the complexity of production environments with external dependencies (e.g., JCL, CICS, VSAM). They also note that the study isolates orchestration strategy under controlled conditions, and results may vary with different prompting strategies or model architectures.

In conclusion, the study posits that embedding LLMs within deterministic execution frameworks is crucial for maintaining economic viability and cost predictability in enterprise-scale modernization, challenging the assumption that delegating all control to the agent is inherently superior.