Comparative Analysis of Rule-Based, Deep Learning, and LLM Approaches for Causality Extraction: A Unified Benchmark

In the vast landscape of human communication, we constantly weave stories of cause and effect. We say that heavy rain caused a river to flood, or that a missed deadline led to a project's failure. Sometimes these connections are obvious, marked by words like "because" or "resulted in." But often, the link is hidden. A text might describe a sequence of events where the cause is never named, leaving the reader to infer the relationship based on context and common sense. For computers to truly understand our language, they must learn to spot these invisible threads, a task known as causality extraction. This is not just a linguistic puzzle; it is a practical necessity for machines that need to analyze news reports, medical records, or safety investigations to understand what happened and why. For years, researchers have tried to teach computers this skill using three different strategies: rigid sets of human-made rules, complex neural networks that learn from examples, and the newest generation of massive language models that can reason through instructions. However, because each group tested their methods on different data with different scoring systems, it has been impossible to know which approach is truly the best or where they fail.

A researcher at the Technical University of Munich decided to settle this uncertainty by building a single, unified testing ground. They gathered existing collections of text from news, medicine, finance, and safety reports, and carefully rewrote the labels for every cause-and-effect relationship so that they all followed the same strict definition. This created a fair playing field where they could test the three main strategies side by side under identical conditions. They pitted simple rule-based systems, which look for specific words and grammar patterns, against deep learning models that have studied thousands of examples, and finally against large language models that can process vast amounts of text and follow complex instructions. The goal was to see not just which method got the highest score, but how each one handled the tricky parts of the task, such as finding causes that are not explicitly stated or connecting events that are far apart in a long document.

The results revealed a landscape that is far more nuanced than the simple story of "newer is better" would suggest. The researcher found that the old-fashioned rule-based systems, which rely on spotting words like "caused" or "due to," are still surprisingly effective when the cause is clearly marked. In these straightforward cases, they perform nearly as well as the most advanced neural networks, but they do so without needing any training data and with almost no computing power. However, these rigid systems hit a hard wall when the cause is hidden. Because they rely entirely on surface-level clues, they fail almost completely when a text implies a relationship without using specific connecting words.

The deep learning models, which are trained on annotated examples, bridged this gap significantly. They proved much better at spotting hidden causes and understanding context within a single sentence or a short paragraph. Yet, they struggled when the cause and effect were separated by several sentences or when the text described a long chain of events. The most powerful tools, the large language models, showed the best overall ability to find these hidden and distant connections. When these models were carefully tuned with specific examples, they outperformed the other methods, particularly in the hardest category of finding causes that are both implicit and spread across a document. However, this superior performance came with a heavy price. The large models required vastly more computing power to run and took significantly longer to process each document than the simpler models. Furthermore, even the best large models made a specific type of mistake: they would often identify the correct idea but get the exact boundaries of the phrase slightly wrong, a problem known as boundary drift.

Perhaps the most important discovery was that no single method solved the problem completely. While the large language models were the strongest overall, they still failed to fully reconstruct the complete causal structure of a long document about half the time. The researcher observed that all three approaches struggled with causal chains where an intermediate event was not explicitly labeled as a cause or effect, suggesting that the current way of defining these tasks might be too simple for the complexity of real-world narratives. The study also showed that while the large models were robust enough to handle a shift from one topic to another, such as moving from medical texts to safety reports, they still lost significant accuracy compared to their performance in their home domain.

The final picture is one of complementary strengths rather than a clear winner. The researcher concluded that the choice of method should depend entirely on the specific needs of the application. If a system needs to process thousands of reports quickly and the causes are usually obvious, a simple rule-based system or a standard neural network might be the most practical choice. But if the goal is to understand complex, hidden relationships in diverse and difficult texts, the large language models are worth the extra cost, provided their limitations are managed. The study suggests that the future of this field lies not in waiting for one technology to defeat the others, but in building hybrid systems that combine the speed and precision of rules with the reasoning power of large models. By acknowledging that each approach has blind spots, researchers can now move toward building tools that are truly capable of understanding the full story of cause and effect.