Reversing Arrows in Large Language Models
This paper presents the first systematic study of how large language models handle inverse relation directionality, revealing systematic asymmetries in their classification performance and demonstrating that model accuracy is sensitive to entity representations while relation descriptions do not consistently improve results.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are teaching a robot to understand the world, not just by reading words, but by connecting them into a giant, invisible web of facts. This is the world of Knowledge Graphs, where every connection has a specific direction, like a one-way street. If you know that "Alice is the mother of Bob," the robot must also understand that "Bob is the child of Alice." But here's the tricky part: if you flip the order and ask about "Bob to Alice," the meaning changes completely. It's like knowing that a key opens a lock doesn't automatically tell you that the lock opens the key. Scientists call these inverse relations, and they are the difference between getting a fact right and getting it backwards.
Recently, we've built super-smart computers called Large Language Models (LLMs). These are like digital brains that have read almost everything on the internet. They are amazing at writing stories and answering questions. But there's a nagging worry: do these digital brains actually understand the direction of these relationships, or are they just guessing based on how the words usually appear together? If a robot thinks "mother" and "child" are just interchangeable labels, it could build a broken map of the world. This paper dives into that exact question, testing whether these AI brains can truly tell the difference between a parent and a child when the order of the names is swapped.
The Great Arrow Reversal Test
In this study, the researchers, Sefika Efeoglu and Adrian Paschke, decided to put five popular AI models to the ultimate test. They didn't just ask the AI to write a sentence; they set up a giant multiple-choice quiz with 5,457 different scenarios. Imagine a game show where the host reads a sentence like, "Niels Bohr and his son Aage arrived..." and asks the AI: "Is the relationship from Niels to Aage 'father' or 'child'?" Then, in the next round, they flip the script: "What is the relationship from Aage to Niels?"
The researchers wanted to see if the AI would get confused when the arrows pointed the other way. To make sure the AI wasn't just relying on memorized patterns (like knowing Niels Bohr is a real scientist), they played a few tricks. Sometimes, they swapped the real people for fake names like "Devin Rodriguez" or "Angela Bradley." Other times, they covered the names completely with "XXX" and "YYY," forcing the AI to rely only on the logic of the sentence, not the fame of the people involved. They also tried giving the AI a reference sheet with definitions of the words, just to see if that helped.
What the AI Actually Did (and Didn't Do)
The results were a bit of a mixed bag, revealing that these AI brains are still a bit clumsy with direction.
First, the study found that direction matters a lot. The AI models were not consistent. Sometimes they were great at figuring out "Head-to-Tail" (like Parent to Child), but when the researchers flipped it to "Tail-to-Head" (Child to Parent), the scores dropped or jumped wildly depending on the model. It's as if the AI is good at reading a map going north but gets lost when asked to drive south. For example, on one dataset, one model got 67.28% right when going one way, but another model only got 2.03% right on the same task in the opposite direction. This suggests that the AI isn't truly "understanding" the relationship; it's often just reacting to patterns it has seen before.
Second, the researchers tested if giving the AI a dictionary definition (a "relation description") would help. They hoped that if they told the AI, "A 'mother' is a female parent," it would do better. The results suggested that this didn't consistently help. In some cases, the extra words made the AI slightly better, but in many others, it didn't change anything or even made things worse. It turns out that just reading a definition doesn't magically fix the confusion about which way the arrow points.
Finally, the "fake name" experiment was the most revealing. When the AI had to deal with real people, it performed reasonably well, likely because it had seen those names and relationships in its training data. But when the researchers swapped in synthetic names or masked the names entirely, the performance became very sensitive. Some models crashed and burned, while others surprisingly held their ground. This suggests that the AI's success often depends on familiarity with the entities rather than pure logic. If the AI recognizes "Niels Bohr," it might guess the answer correctly just because it knows the story, not because it understands the concept of "fatherhood."
The Bottom Line
The paper concludes that while Large Language Models are powerful, they haven't quite mastered the art of inverse relation directionality. They show systematic asymmetries, meaning they are often better at one direction than the other, and they seem to rely heavily on knowing the specific people involved rather than the logic of the relationship itself.
The authors are careful to note that they haven't "solved" the problem or proven that the AI is broken forever. Instead, their work suggests that we need to be more careful when using these models for tasks that require precise, directional knowledge. If we want AI to build accurate maps of the world, we can't just assume it understands that "mother" and "child" are two sides of the same coin. We have to keep testing, keep flipping the arrows, and make sure the robot isn't just guessing based on who it thinks is famous.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.