The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages
This paper investigates how coding agents exhibit significant and consistent variations in token consumption across different programming languages, revealing that inefficiencies stem from behaviors such as generating non-compiling code in unfamiliar languages, distrusting provided tests, and prototyping in Python, thereby establishing by-language token efficiency as a critical metric for benchmarking and cost optimization.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have hired a brilliant, tireless robot assistant to write computer programs for you. You give it a puzzle, and it starts typing code, running tests, fixing mistakes, and trying again until it gets it right. This is the world of "coding agents," a new kind of artificial intelligence that doesn't just answer questions but actually builds software. But here's the catch: these robots don't work for free. Every time they think, type a word, or check their work, they consume "tokens." Think of tokens like fuel for a car or minutes on a phone plan. The more the robot thinks and rewrites, the more fuel it burns.
Now, imagine you ask this robot to solve the exact same puzzle, but this time you tell it to write the solution in a different language. Maybe it writes in Python (a language that looks a lot like English), or maybe it writes in OCaml (a language that looks more like a secret mathematical code). You might assume the robot burns the same amount of fuel regardless of the language, as long as the puzzle is the same. But what if the language itself makes the robot trip over its own feet? What if some languages are like smooth highways for the robot, while others are like muddy swamps where it gets stuck, spins in circles, and burns extra fuel just trying to figure out how to walk? This is the question a team of researchers set out to answer: Does the programming language you choose change how much "fuel" (tokens) your AI assistant burns to get the job done?
The researchers, Zixuan Wu, Carolyn Jane Anderson, and Arjun Guha, decided to investigate this by turning their coding agents loose on a massive set of 100 programming puzzles. They didn't just look at whether the robots got the right answer; they watched every single step the robots took, like a detective reviewing security footage. They tested five different AI models on four different languages: Python, Java, Rust, and OCaml.
Their biggest discovery was that the language matters a huge amount. Even when the puzzles were exactly the same difficulty, the robots burned significantly more tokens when working in OCaml and Rust compared to Python. In fact, for some models, writing in OCaml cost nearly 1.7 times more tokens than writing in Python. The authors suggest that this isn't because the final code was longer, but because the robots got confused, made more mistakes, and had to try harder to fix them in the less familiar languages.
But the real fun was in watching how the robots got stuck. The researchers found that when the robots hit a wall in a tricky language like OCaml, they didn't just give up. Instead, they started acting strangely. Sometimes, they would get a solution that passed all the tests, but then they would keep rewriting it anyway, adding unnecessary comments or changing the style, as if they couldn't trust their own success. Other times, they would get stuck on a simple syntax error (like a missing semicolon) and spin in circles, rewriting the same broken code over and over again without realizing what was wrong.
Perhaps the most amusing behavior was that when the robots encountered a language they didn't know well, they would cheat a little. They would secretly solve the problem in Python first—where they felt comfortable and confident—and then try to translate their Python solution into the target language, like OCaml or Rust. It's like a student who knows how to solve a math problem in their head but then tries to write it down in a language they barely speak, getting confused by the grammar even though they know the math.
The study also looked at a concept called "tokenmaxxing," which the authors jokingly describe as a way to make an AI spend as much money as possible. They found that if you really wanted to burn through your budget, you should pick OCaml. The robots would struggle, rework, and overthink their way through the problems, consuming a massive amount of tokens. On the flip side, if you want your AI to be efficient and cheap, Python is the clear winner.
In the end, the paper suggests that the choice of programming language isn't just about what the code looks like; it changes how the AI thinks. Some languages act like a gentle guide, while others act like a maze that makes the robot spin in circles. The authors hope that by understanding these behaviors, we can build better, more efficient robots that don't waste so much fuel, no matter what language they are asked to speak. They also warn that while they found these patterns in their specific experiments, future robots might get smarter and handle these tricky languages better, so the results are a snapshot of how things work right now, not a permanent rule of the universe.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.