Predicting semantic consistency between images and Korean hashtags on Instagram
This paper proposes a five-level ordinal classification framework for predicting semantic consistency between images and Korean Instagram hashtags, demonstrating that a fine-tuned Korean-adapted CLIP model outperforms both its English counterpart and classical classifiers on a GPT-4o-annotated dataset, while highlighting the intrinsic ambiguity of the task and the critical importance of linguistic suitability in multimodal understanding.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
On platforms like Instagram, a photograph is rarely just a picture. It is a statement, a mood, and a piece of cultural conversation all at once. When a user posts a photo of a simple cup of coffee, they might tag it with a literal description like "morning brew," but more often, they use a hashtag that conveys a feeling, such as "healing" or "Monday blues." These tags do not merely describe what is visible; they express emotion, slang, and shared cultural understanding. For a computer to truly understand social media, it must learn to read these subtle, non-literal connections between an image and the words attached to it. This is a difficult task because the relationship between a photo and its hashtag is not a simple yes-or-no match. Instead, it exists on a spectrum, ranging from completely unrelated to deeply meaningful, and this meaning often relies on context that a machine cannot easily see.
Researchers at Seoul National University and Sungshin Women's University set out to teach computers how to navigate this complex landscape. They focused specifically on Korean Instagram posts, where language is filled with abbreviations, memes, and culturally specific expressions that can be tricky even for fluent speakers. The team treated the problem not as a binary choice, but as a five-level scale. They asked their models to decide if a hashtag was highly inconsistent, low, medium, high, or highly consistent with the image. To do this, they built a dataset of over 12,000 image and hashtag pairs from public posts. Because manually labeling thousands of subjective connections is slow and prone to human disagreement, they first used a powerful artificial intelligence tool to generate initial labels based on strict guidelines. They then validated these labels by having four human experts review a sample of the data, ensuring the computer's judgments were close enough to human understanding to be useful.
The researchers tested several different approaches to see which method worked best. They started with existing, general-purpose models that had been trained on millions of image and text pairs from the internet. These models were good at matching pictures to literal descriptions, like a dog with the word "dog," but they struggled with the abstract, emotional tags common on social media. When the team simply measured the mathematical distance between the image and the text in the computer's memory, the results were poor. The models could not distinguish between the five levels of consistency. However, when the researchers added a simple decision-making layer on top of these frozen models, the performance improved. This showed that while the base models understood the general content, they needed a specific guide to interpret the graded, subjective nature of the tags.
The most significant discovery came when the team tried to teach the models specifically for this task by adjusting their internal settings, a process known as fine-tuning. They tested two different base models: one designed for general use and another specifically adapted for the Korean language. The results were striking. When they fine-tuned the general model, its performance actually got worse. It seemed that trying to force a general-purpose system to learn a specific, culturally nuanced task without a strong foundation in that language caused it to lose its way. In contrast, the Korean-adapted model thrived when fine-tuned. It achieved the highest accuracy and, more importantly, became much better at identifying the difficult, less common tags, rather than just guessing the most frequent ones. This suggested that having a model that already understood the language and culture was a prerequisite for successfully teaching it to understand the nuance of social media expression.
To see if this improvement was truly necessary, the researchers compared their best fine-tuned model against powerful, traditional computer programs that simply looked at the frozen data without changing the model itself. They found that these traditional programs could match the fine-tuned model's overall accuracy. However, the fine-tuned model was superior in one crucial way: it was more balanced. It did not ignore the rare or difficult cases. While the traditional programs were good at getting the easy, common answers right, they often failed on the edge cases that matter most for filtering out irrelevant content. The fine-tuned model, having learned to adjust its own internal understanding of the images and words together, provided a more stable and fair assessment across all types of tags.
The study also highlighted the inherent difficulty of the task itself. Even among the human experts who validated the data, there was no perfect agreement. They only matched each other's exact scores on about half of the pairs, with most disagreements being just one step apart on the five-level scale. This suggests that the ambiguity is not just a failure of the computer, but a feature of human communication. When a hashtag expresses a mood or a cultural joke, there is often no single "correct" answer. The best computer model managed to get the right answer or a neighboring answer about 72 percent of the time. This level of performance, while not perfect, represents a significant step forward in teaching machines to understand the emotional and cultural layers of human expression, moving them closer to a point where they can truly grasp the full meaning of a social media post.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.