🤖 machine learning

Is Code Better Than Language for Algorithmic Reasoning

This paper demonstrates that for tool-augmented language models, the performance advantage of code over natural language in algorithmic reasoning stems primarily from reliable external execution rather than the intermediate code representation itself, as code-simulated reasoning yields no significant gain over natural-language reasoning.

Original authors: Terry Tong, Yu Feng, Surbhi Goel, Dan Roth

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Terry Tong, Yu Feng, Surbhi Goel, Dan Roth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a very tricky math puzzle. You have two main ways to get the answer:

The "Talker" Method: You ask a smart assistant to think out loud in plain English, step-by-step, and then tell you the answer.
The "Coder" Method: You ask the same assistant to write a computer program to solve the puzzle, and then you let a computer actually run that program to get the answer.

For a long time, people noticed that the "Coder" method usually worked much better. But nobody was sure why. Was it because writing code forces the assistant to think more clearly? Or was it because letting a computer run the code is just more reliable than trusting the assistant to do the math in its head?

This paper sets up a clever experiment to figure out which factor is the real hero.

The Three-Route Experiment

The authors created a "three-lane highway" to test this, using a 40-task benchmark of algorithmic puzzles (like sorting lists, finding paths, or doing complex math).

Route 1 (The Pure Talker): The assistant solves the problem entirely in English. It thinks, writes a paragraph of reasoning, and gives the answer.
- Result: It got about 17% of the answers right.
Route 2 (The Fake Coder): The assistant writes the code (like Python), but then pretends to run it. Instead of giving the code to a computer, the assistant reads its own code and simulates the steps in its head, writing the result in English.
- Result: It got about 17% of the answers right.
- The Big Reveal: This is almost exactly the same as Route 1. Writing the code didn't help the assistant think better; it just changed the format of its thoughts.
Route 3 (The Real Coder): The assistant writes the exact same code as in Route 2, but this time, it hands the code to a real computer (a Python runtime) to execute.
- Result: It got about 49% of the answers right.

The "Why" Behind the Magic

The paper uses a few creative ways to explain these results:

1. The "Translation" Analogy (Representation vs. Execution)
Think of the "Trace" (the reasoning steps) as a recipe.

Route 1 is a recipe written in a long, flowery story.
Route 2 is the same recipe, but written in a strict, structured list.
Route 3 is that strict list, but instead of a human chef trying to read it and guess the outcome, you hand it to a robot chef that follows the instructions perfectly.

The experiment showed that changing the recipe from a story (English) to a list (Code) didn't make the human chef any smarter. The human chef (the LLM) made the same mistakes whether reading the story or the list. The massive jump in performance only happened when the robot chef (the computer executor) took over.

2. The "Nuisance" Theory
The authors argue that natural language is full of "noise." You can say the same thing in a thousand different ways ("add the numbers," "sum them up," "put them together"). This extra variety confuses the model. Code is stricter; it has fewer ways to say the same thing.
However, the paper proves that even though code is "cleaner," the model still can't use that cleanliness to solve the math problems on its own. The code doesn't magically give the model new math skills.

3. The "Recovery" Test
The authors looked at the cases where the computer got the answer right, but the human assistant (simulating the code) got it wrong. This happened 33% of the time.
Conversely, they looked at cases where the computer failed (maybe the code was broken), but the human assistant guessed the right answer anyway. This happened only 1.6% of the time.
This proves that the computer is a much more reliable "executor" than the human assistant is at "simulating" the code.

The Bottom Line

The paper concludes that Code is not better than Language because it's a better way to think.

The advantage of using code isn't that it forces the AI to be smarter. The advantage is that Code allows the AI to hand the work off to a machine that doesn't make math errors.

The Bottleneck: The problem isn't that the AI can't write good code; it's that the AI is bad at checking its own work or doing the math in its head.
The Solution: The real power comes from the tool (the computer running the code), not the language (the code itself).

In short: If you want an AI to solve a hard math problem, don't just ask it to write code and hope it figures out the answer. Ask it to write code, and then let a computer actually run it. That is where the magic happens.

Technical Summary: Is Code Better Than Language for Algorithmic Reasoning?

Problem Statement

In tool-augmented language model (LLM) systems, it is well-established that translating problems into solver-executable code (delegating execution to an external tool) often outperforms end-to-end natural language (NL) reasoning on complex algorithmic tasks. However, the source of this performance gain remains ambiguous. Prior comparisons conflate two distinct factors:

Intermediate Representation: The shift from natural language traces to code traces.
Execution Mechanism: The shift from LLM-based simulation (forward pass) to deterministic external execution (e.g., Python runtime).

Because previous studies compared "NL reasoning" directly against "Code + External Execution," they could not isolate whether the improvement stemmed from the structured nature of code, the reliability of the external executor, or both. This paper addresses the research question: Is code representation inherently superior to natural language for algorithmic reasoning, or is the advantage primarily driven by reliable external execution?

Methodology: The Three-Route Framework

To disentangle representation from execution, the authors propose a systematic Three-Route Framework that decomposes the reasoning pipeline into two stages: Trace Generation and Execution.

Route 1 (Direct NL): The LLM generates a natural language Chain-of-Thought trace and subsequently uses the same LLM to simulate the reasoning and produce an answer.
- Trace: Natural Language.
- Executor: LLM (NL simulation).
Route 2 (Code + NL Simulation): The LLM generates a code trace (e.g., Python), but instead of executing it externally, the same LLM simulates the code execution in natural language to produce an answer.
- Trace: Code.
- Executor: LLM (Code simulation).
Route 3 (Code + Solver Execution): The LLM generates the same code trace as Route 2, but the code is executed by a deterministic external Python runtime.
- Trace: Code.
- Executor: External Solver (Deterministic).

Experimental Setup:

Benchmark: A 40-task verifiable algorithmic benchmark covering arithmetic, dynamic programming, graph algorithms, string manipulation, geometry, sorting, and NP-hard optimization.
Data: 1,113 unique problem instances across 3 random seeds, evaluated on 6 models (including closed-source like GPT-4o-mini and open-source like Mixtral-8x22b).
Metrics: End-task accuracy under 0–1 loss, analyzed via paired statistical tests (McNemar) and cluster-bootstrap confidence intervals.

Key Contributions

1. Empirical Disentanglement of Factors

The framework allows for pairwise comparisons that isolate specific variables:

Route 1 vs. Route 2 (Isolating Representation): By holding the LLM executor constant and only changing the trace modality (NL vs. Code), the authors test if code representation itself offers an advantage.
Route 2 vs. Route 3 (Isolating Execution): By holding the code trace constant and only changing the executor (LLM simulation vs. Python runtime), the authors test if the execution mechanism is the driver of performance.

2. Theoretical Formalization (Decision-Theoretic Model)

The authors formalize the intuition that code is a "canonicalization" of natural language. They propose a linear model where:

Both traces share a core ( $B$ ) containing decision-relevant algorithmic information (e.g., recurrence relations).
Natural language traces contain additional nuisance variation ( $\eta_N$ ) such as paraphrasing, verbosity, and stylistic choices that do not alter the answer.
Theorem: Under this model, code is representation-non-inferior to natural language. The extra nuisance in natural language increases the risk (error) for finite-sample learners without providing additional signal. Thus, changing the representation alone should not yield significant gains if the core information is preserved.

3. Reconstruction Intervention

To validate the theory empirically, the authors perform a "reconstruction intervention." They take code traces generated by the model and translate them back into natural language using a fixed transformation. They find that these translated NL traces recover performance comparable to native NL traces. This confirms that code retains all decision-relevant information and that the "loss" in NL reasoning is not due to missing algorithmic content but rather the presence of nuisance variation.

Results

Performance Comparison

On the 40-task benchmark, the results are as follows:

Route 1 (NL + NL): 17.21% accuracy.
Route 2 (Code + NL Sim): 17.37% accuracy.
Route 3 (Code + Python Exec): 48.84% accuracy.

Statistical Findings:

Route 1 vs. Route 2: The difference is +0.15 percentage points (pp), with a 95% confidence interval of $[-0.30, +0.61]$ pp ( $p=0.39$ ). This indicates no statistically significant difference between natural language reasoning and code simulation when the executor is the same LLM.
Route 2 vs. Route 3: The difference is +31.47 pp, with a 95% CI of $[+29.20, +33.71]$ pp ( $p < 0.001$ ). This indicates a massive, statistically significant advantage for deterministic external execution over LLM simulation, even when the code trace is identical.

Difficulty Scaling

As task difficulty increases (controlled by parameters like digit length or matrix size), Route 3 maintains high accuracy while Routes 1 and 2 degrade significantly.

Recovery Analysis

The authors analyze cases where Route 2 (simulation) succeeds but Route 3 (execution) fails ("recovery mass").

Recovery Mass: 1.61% (Cases where the LLM correctly simulates flawed code or ignores execution failure).
Execution-Win Mass: 33.08% (Cases where execution succeeds but the LLM simulation fails).
This confirms that the LLM rarely compensates for execution errors; conversely, the external solver consistently outperforms the LLM's ability to simulate the code.

Significance and Claims

The paper claims that in the evaluated setting of algorithmic reasoning:

Representation is not the bottleneck: The intermediate intervention of using code traces does not meaningfully improve performance over natural language traces when the execution mechanism remains an LLM. The performance gap between Route 1 and Route 2 is negligible.
Execution is the bottleneck: The primary driver of the performance gains observed in tool-augmented systems is the reliability of the external executor, not the structured nature of the code representation itself.
Mechanism of Tool Use: The advantage of code-based tool use arises because it enables the hand-off of a generated trace to a deterministic solver. The code representation serves primarily as a vehicle to enable this reliable execution, rather than acting as a superior reasoning substrate for the LLM itself.

The authors conclude that for algorithmic tasks with verifiable answers, the central advantage of tool use is the ability to delegate execution to a deterministic environment, rather than the inherent superiority of code as a reasoning format over natural language.