NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection

In the digital age, a lie often travels faster than the truth, but not always because the truth is hidden. Sometimes, the lie is built from the truth itself. This is the nature of out-of-context misinformation, a deceptive practice where a genuine, unaltered photograph is paired with a false or misleading caption. The image remains untouched; a photo of a real crowd, a real building, or a real person is taken from its original moment and place. The deception lies entirely in the story told alongside it. A picture of a flood from ten years ago might be presented as a current disaster, or a photo of a politician in one country might be claimed to show them in another. Because the image itself is authentic, it passes the visual checks that usually catch doctored photos or deepfakes. The challenge for computers and humans alike is not to spot a fake image, but to realize that the story attached to it does not match the reality captured in the frame.

For years, researchers have built tools to catch these mismatches, but most of their work has focused on English-language news and high-resource settings. This left a significant gap in understanding how these deceptions work in other parts of the world, particularly in regions where languages and cultures are distinct. In Nepal, where misinformation can inflame political tensions or spread panic during natural disasters, the problem is urgent. The local context is vital; a caption might reference a specific village, a local official, or a regional event that an English-speaking system would never recognize as suspicious. To address this, a researcher named Sanjeev Khatiwada set out to build a new kind of test bed, one designed specifically for the Nepali language and the unique ways misinformation spreads there.

The result is a new collection of data called NepOOC, the first public benchmark of its kind for Nepali. It contains 1,090 pairs of images and captions, split evenly between honest posts and deceptive ones. The deceptive posts were carefully categorized into five types of lies: completely made-up stories, captions that are factually wrong but not about time or place, images from the wrong time, images from the wrong location, and images of the wrong people. To ensure the data was reliable, multiple experts reviewed the pairs, and they agreed on the classification of the deceptive posts with a high degree of consistency. The dataset includes real-world examples drawn from fact-checking reports, news portals, and social media archives, capturing the messy reality of how these lies actually appear in the wild.

With this new dataset in hand, the researcher tested five different computer systems to see which could best spot the lies. These systems ranged from simple text readers to complex machines that tried to look at both the picture and the words at the same time. The goal was to see if looking at the image helped the computer understand the lie, or if reading the caption was enough. The findings were surprising and clear. The best-performing system was a multimodal model that combined a visual analyzer with a text reader, but it did not perform better than a system that only read the text. In fact, a model that looked only at the words achieved the exact same high score as the most complex system that looked at both the image and the words.

The data showed that the visual part of the image provided almost no help in solving the puzzle. When the researchers tested systems that looked only at the pictures, ignoring the text entirely, they performed no better than random guessing. The pictures themselves, being authentic, held no clues to the deception. The lie was entirely in the story. The text-only system, which read the caption and checked it against its knowledge of the world, was able to identify the mismatch with nearly perfect accuracy. This suggests that for this specific type of problem, at this current scale of data, the words carry all the necessary information. The visual features, which are so often the focus of artificial intelligence research, were essentially silent in this context.

The study also explored whether making the computer systems more complex or specialized would lead to better results. They tested different types of neural networks and tried to adapt models specifically for the Devanagari script used in Nepali. The results indicated that these architectural tweaks did not make a significant difference. A standard, well-trained text reader worked just as well as a specialized one. The most important factor for success was not the sophistication of the machine, but the amount of data it had to learn from. When the researchers tested the systems with smaller amounts of training data, the more complex models struggled, while the simpler text-based models remained steady. This points to a clear path forward: the key to better detection is not building smarter algorithms, but gathering more examples of these lies to teach the systems.

This work challenges a common assumption in the field of artificial intelligence, which often assumes that combining vision and language always leads to better performance. In the case of out-of-context misinformation in Nepal, the visual component was redundant. The deception was a linguistic one, rooted in the specific claims made about people, places, and events. The study concludes that for low-resource languages and specific types of misinformation, focusing on the text and expanding the dataset is a more direct and effective path to progress than trying to engineer more complex multimodal systems. It highlights that sometimes, the most powerful tool is simply a better understanding of the story being told.