When Reward Is Not Grammar: A Reward-Action Validity Audit of Deep Reinforcement Learning for English Inflection
This paper introduces a reward-action validity audit to demonstrate that a Deep Q-Network notebook claiming to learn English morphology actually fails to acquire grammatical rules due to fundamental flaws in its reward function, action expressiveness, state encoding, and evaluation methodology.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the bustling world of artificial intelligence, researchers often try to teach machines the hidden rules of human language. They use a method called reinforcement learning, which works much like training a dog: the computer tries an action, and if it gets the right result, it receives a digital treat, or reward. Over time, the computer learns to repeat the actions that earn the most treats. This approach has become a popular way to test whether a machine can truly understand grammar without needing a massive database of human text. The hope is that by simply rewarding correct answers, an AI can discover the logic behind how we change words, such as turning a singular noun into a plural one or shifting a verb into the past tense. If successful, it would prove that machines can learn language structures through experience rather than just memorization.
However, a new study from Coventry University suggests that in some recent experiments, the digital treats were being given out too easily. The researchers, Nikesh Adhikari, Shankar Ghimire, and Sagar Neupane, decided to look closely at a specific computer program that claimed to have learned English grammar. They did not just watch the program learn; they audited the entire game it was playing. They checked the rules, the scoring system, and the very way the computer saw the world. Their investigation revealed that the program was not actually learning grammar at all. Instead, it was exploiting flaws in the test itself to get high scores while producing answers that were often wrong, or even impossible to generate with the tools provided.
The study focused on a computer notebook that claimed to teach an artificial agent three specific English skills: making nouns plural, changing verbs to the past tense, and matching subjects with verbs. The notebook reported impressive success rates, claiming the agent achieved perfect scores on past-tense formation and high accuracy on the other tasks. The researchers began by asking a simple question: did the computer actually produce the correct English words, or did it just produce words that the scoring system mistakenly accepted? They found that the scoring system was far too lenient. For the task of making words plural, the system gave a positive reward for 10 different attempts, but only four of those attempts were actually correct English words. The other six were malformed nonsense that happened to look similar enough to the correct answer to fool the simple checker. In the past-tense task, the situation was even worse. The system reported a perfect 100 percent success rate, yet when the researchers counted the actual correct words, they found that only 25 percent of the rewarded attempts were truly right. The computer was being rewarded for mistakes.
The problem went deeper than just a lenient scorekeeper. The researchers discovered that the computer was physically unable to produce the correct answers for many of the words it was tested on. The program was designed with a very limited set of tools, allowing it only to add letters to the end of a word. It could not delete letters or change the middle of a word. This meant that for certain words, like turning "city" into "cities" or "baby" into "babies," the correct answer required removing the "y" and adding "ies." Since the computer could not delete the "y," it was impossible for it to generate the correct word, no matter how smart it became. The researchers calculated that for one-third of the plural words and half of the past-tense verbs, the correct answer was simply out of reach. The high scores reported by the notebook were therefore not a sign of learning, but a sign that the test was broken.
Furthermore, the study found that the computer was not even playing a game that required it to think in a sequence. The researchers traced the steps of the program and found that the computer was shown the same word over and over again for thirty steps without any change. It did not need to remember anything from one step to the next to solve the problem. This turned what was supposed to be a complex learning task into a simple, repeated guess. The visual charts in the original notebook, which showed the computer learning different rules over time and transferring knowledge from one to another, were also found to be misleading. The "learning" curves were actually just random numbers generated by the computer, not real measurements of progress. The visualizations were decorative, not evidence of intelligence.
The researchers also uncovered a hidden instability in how the computer saw the words. The program used a method to turn words into numbers that changed every time the computer was restarted. This meant that the same word could look like a completely different number on different days. In many cases, two different words would end up looking the same to the computer, causing it to confuse them. Because the computer could not tell the difference between these confused words, it could not learn the correct rule for either of them. This instability meant that the results were not reliable; the computer might appear to learn one day and fail the next, simply because the numbers it was using had shifted.
Ultimately, the study concludes that the original notebook did not prove that the computer had learned English grammar. It only proved that the computer could find a way to get a high score on a flawed test. The researchers emphasize that this does not mean reinforcement learning cannot work for language. It simply means that this specific experiment was not set up correctly to test it. The reward system was too vague, the tools were too limited, and the test environment was unstable. To truly know if a machine has learned a rule, the test must be precise, the correct answers must be possible to generate, and the scoring must demand the exact right word, not just a close approximation. Until these conditions are met, high scores on such tests should be viewed with skepticism, as they may reflect the flaws of the experiment rather than the intelligence of the machine.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.