Imagine you are trying to teach a brilliant, well-read robot how to play a game. You give it a library of every book ever written, and it can recite poetry, solve complex math problems, and write code. But when you sit it down at a board for a simple game like Tic-Tac-Toe, it starts making silly mistakes. It makes a move that looks good on the surface but loses the game five turns later. This is the world of Large Language Models (LLMs): super-smart AI that can talk and reason, but sometimes struggles when it has to make a series of decisions where one bad move ruins everything.

The big question researchers are asking is: Why does this happen? Is the robot bad at thinking through the steps in the moment? Or is it just "hallucinating" because it's trying to remember a game it saw in a book but doesn't actually understand the logic? To find out, scientists use sequential decision-making, which is just a fancy way of saying "making a chain of choices where the result of one choice changes the board for the next." They test this in zero-sum games, like chess or checkers, where one player's win is the other's loss. These games are perfect for testing because the rules are strict, and we can use math to know exactly what the "perfect" move is. If a robot can't beat a perfect computer program at Tic-Tac-Toe, something is broken in its brain.

The Puzzle of the Reasoning Bottleneck

In this study, researchers from the Czech Technical University in Prague decided to poke the bear. They took the smartest AI models available—ranging from efficient, smaller models to "frontier" giants like GPT-5.4 and Gemini 3.1 Pro—and challenged them to play simple games: Tic-Tac-Toe, Nim (a game with piles of stones), and Connect Four.

The results were a bit embarrassing for the AI. Even the most advanced models, which can solve competition-level math problems, played sub-optimally. In Tic-Tac-Toe, where a human child can learn to never lose in an afternoon, these AI models made four to twelve mistakes per hundred moves. In Connect Four, the "frontier" models lost almost every game against a standard computer opponent, often collapsing within the first ten moves of a game that should last forty.

The researchers wanted to know: Is the AI just forgetting the game? Maybe it's trying to recall a strategy it saw in its training data, but the game looks slightly different, so it gets confused. To test this, they played a trick on the AI. They kept the game exactly the same underneath but changed the "skin" on top. They turned Tic-Tac-Toe into a "Magic Square" where you win by picking three numbers that add up to 15, or they told a "Story" about crossing a river.

Here is the twist: The AI didn't get much better or worse. Even when the game was disguised as a math puzzle or a story, the AI still made the same number of mistakes. This suggests the problem isn't that the AI is forgetting a memorized strategy or failing to understand the rules; it's that the AI has a reasoning bottleneck in the moment. It's like a student who can recite a history textbook and understands the logic perfectly, but freezes when asked to solve a simple logic puzzle on the spot because they can't connect the current situation to the right next step. The gap between what the AI knows and what it can do in a changing environment is real, and it's stubborn.

The Solution: A "Reflective" Notebook

Since retraining these massive AI models is expensive and risky (it might make them forget how to write code or speak), the researchers asked: Can we help the AI without changing its brain?

They built a new framework called REAPER (Reflective Experiential Agent with Periodic Extraction of Rules). Think of REAPER not as a new brain, but as a smart notebook and a strict coach sitting next to the AI.

Here is how it works:

1. The Game: The AI plays a game against an opponent.
2. The Reflection: After the game ends, the AI doesn't just look at the final score (Win/Loss). Instead, the REAPER system forces it to look at every single move it made. It asks: "Was this move good for the board right then? Did it help me win, or did it hurt me?"
3. The Credit Assignment: This is the magic part. In a normal game, if you lose, you might think every move was bad. But REAPER says, "Wait, you made a great move in the middle, even though you lost in the end." It gives credit for good moves and blame for bad ones, even if the final result was a loss.
4. The Rule Book: Every few games, REAPER takes all these reflections and writes them down as simple, natural-language rules. Instead of just remembering "Move X was good in this specific spot," it learns "If the opponent threatens a corner, block it."
5. The Loop: The AI plays again, but this time it reads its own notebook of rules and past mistakes before making a move.

The Results: Smarter, Not Stronger

The researchers tested this on Tic-Tac-Toe using a smaller, efficient AI model (GPT-5 nano). The results were clear:

• The Baseline: Without the notebook, the AI played poorly, making many mistakes.
• The "Original" Attempt: They tried a simple version of the notebook, but it was too messy and the AI kept making formatting errors.
• The REAPER Success: With the full system of reflection and rule extraction, the AI's performance jumped. It started drawing games (the best possible outcome against a perfect opponent) about 86.8% of the time, compared to 81.8% for the improved baseline.

Crucially, the AI didn't need to be retrained. It learned purely from its own experience, just like a human player who studies their past games to get better. The researchers found that this method didn't just make the AI play better; it actually made it more efficient, using fewer words (tokens) to make its decisions because it had a clear strategy to follow.

What This Means

This paper suggests that the problem with AI in games isn't that they aren't smart enough to understand the rules; it's that they struggle to connect a long chain of actions to a final result. By giving them a way to reflect on their mistakes and extract general rules from specific games, we can turn a clumsy player into a strategic one without touching a single line of their code.

The researchers are careful to note that this is a simulation on simple games. They don't claim this solves all AI problems or that it will work perfectly on complex, real-world scenarios yet. But it proves a vital point: Experience memory and self-reflection are powerful tools. If we can teach AI to look back at what it did, learn from it, and write down the lessons, we might be able to build agents that don't just "think" in a vacuum, but actually learn from the world around them.