💬 NLP

Towards Robust Evaluation of Visual Activity Recognition: Resolving Verb Ambiguity with Sense Clustering

This paper proposes a vision-language clustering framework that resolves verb ambiguity in visual activity recognition by grouping synonymous and perspective-dependent verbs into sense clusters, thereby offering a more robust and human-aligned evaluation method compared to standard exact-match metrics.

Original authors: Louie Hong Yao, Nicholas Jarvis, Tianyu Jiang

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Louie Hong Yao, Nicholas Jarvis, Tianyu Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a student's essay about a picture. The picture shows a person on a bicycle.

If the student writes, "The person is riding a bike," and your answer key says "The person is biking," a strict computer program might mark this as wrong. It sees "riding" and "biking" as two different words, so it gives a zero.

But any human would say, "Wait, that's the same thing! The student is right."

This paper is about fixing that unfair grading system for computers that try to understand what's happening in pictures.

The Problem: The "One Right Answer" Trap

Currently, computers are tested on visual activity recognition (identifying actions in photos) using a method called "Exact Match." This means the computer's guess must be the exact same word as the human label.

The authors argue this is broken because:

Synonyms exist: "Teaching" and "lecturing" describe the same event.
Perspectives change: A picture of a marching band could be described as "marching" (focusing on the legs) or "performing" (focusing on the music). Both are true, but they are different words.

If a computer guesses "lecturing" but the label is "teaching," the computer gets a failing grade, even though it understood the picture perfectly.

The Solution: The "Group Hug" Method

The researchers propose a new way to grade these computers. Instead of looking for one single magic word, they build "Sense Clusters."

Think of it like organizing a messy closet. Instead of trying to find one specific shirt, you group similar shirts together.

Step 1: They take a picture and ask powerful AI models (like GPT-4o and Llama) to describe it using many different verbs.
Step 2: They use a smart sorting algorithm to group these verbs. If "riding," "biking," and "cycling" all appear with the same pictures, they get put in the same "cluster."
Step 3: If a picture belongs to a cluster, and the computer guesses any word inside that cluster, it gets credit.

What They Found

They tested this new grading system on the imSitu dataset (a huge collection of 126,000 pictures with action labels). Here is what they discovered:

The "Four Perspectives" Rule: On average, a single picture can be correctly described by about four different "clusters" of words. For example, a picture of a teacher might have a cluster for "teaching," another for "lecturing," another for "instructing," and maybe one for "educating."
Better Grades for Smart Models: When they re-graded existing computer models using their new "cluster" method, the scores went up significantly.
- Example: A model that got 56% correct with the old method jumped to 72% with the new method.
- Why? The old method punished the model for using a synonym or a different perspective. The new method realized the model was actually smart.
Human Agreement: When the researchers asked real humans to grade the models, the "cluster" method's scores matched the humans much better than the old "exact match" method did. The old method was too harsh; the new method was fair.

Why Not Just Use "Similarity Scores"?

The authors also tested a popular tool called CLIPScore, which tries to measure how similar a word is to a picture using math. They found that while it's good at general things, it struggles with specific action verbs. It's like a judge who says, "This picture and the word 'eating' are 90% similar," even if the person in the picture is actually "cooking." The context gets lost. Their "cluster" method is more precise because it groups words based on how they are actually used in real pictures.

The Bottom Line

The paper doesn't claim this will cure diseases or build self-driving cars immediately. Instead, it offers a better ruler for measuring how good our AI is at understanding pictures.

By realizing that there isn't just one "right" word for an action, but rather a whole family of correct words, we can stop penalizing AI for being creative and start giving it credit for actually understanding the scene. It's like finally telling the student, "Yes, 'riding' is a correct answer for 'biking'."

Technical Summary: Towards Robust Evaluation of Visual Activity Recognition

Problem Statement

Evaluating visual activity recognition systems is fundamentally challenged by inherent ambiguities in verb semantics and image interpretation. Standard evaluation relies on exact-match metrics, where a model's prediction is deemed correct only if it matches a single "gold" verb label. This approach fails to account for two primary sources of ambiguity:

Synonymy: Multiple synonymous verbs (e.g., teaching vs. lecturing) can describe the same event.
Multi-perspectives: A single image can be validly described by distinct verbs reflecting different viewpoints (e.g., marching vs. performing for a band).

Consequently, exact-match evaluation systematically underestimates model performance by penalizing semantically valid but lexically distinct predictions. While metrics like CLIPScore exist, they often rely on generic similarity measures that fail to capture fine-grained verb sense distinctions or shifts in perspective.

Methodology

To address these limitations, the authors propose a vision-language clustering framework that constructs "verb sense clusters." This framework groups images and verbs into coherent clusters where each cluster represents a distinct perspective or sense of an activity. The methodology proceeds in three main stages:

1. Acquiring Image-Verb Pairs

To mitigate labeling bias inherent in the original imSitu dataset (where images were retrieved via verb-based Google searches), the authors query two multimodal Large Language Models (LLMs)—GPT-4o mini and Llama-3.2-90B—to generate all appropriate verbs for each image. They retain only verbs present in the imSitu vocabulary (504 verbs) and construct nodes representing <image, verb> pairs. These nodes are embedded using a multimodal Llama model (Llama-3.2-11B) by processing text and image inputs simultaneously, extracting the final hidden state after the cross-attention layer.

2. Two-Step Clustering Framework

The framework employs a two-step clustering process to group these <image, verb> pairs:

Step 1: Same-Verb Clustering: For each of the 504 verbs, the associated images are clustered to disambiguate fine-grained senses of that specific verb. The optimal number of clusters ( $k$ ) for each verb is determined by maximizing the Silhouette score across a range of $k \in [2, 16]$ . Both K-Means and Hierarchical Agglomerative Clustering (HAC) are tested.
Step 2: Cross-Verb Clustering: The resulting clusters from Step 1 are further clustered to group verbs with shared meanings across different lexical items. The number of final clusters ( $k_r$ ) is determined by a ratio $r$ applied to the total verb count (504), optimized via the Silhouette score.

3. Evaluation Protocol

The resulting clusters serve as the ground truth for evaluation. A model prediction is considered correct if the predicted verb appears in any cluster containing the target image. This is compared against:

Exact Match: Standard gold-label matching.
WordNet-Synset Match: Checking if the prediction shares a synset with the gold label.
Human Judgment: Manual annotation of 100 sampled images to assess alignment.

Key Results

Clustering Quality and Ambiguity Analysis

Cluster Structure: Analysis reveals that, on average, each image maps to approximately 4 sense clusters, and each cluster contains roughly 2 verbs.
Ambiguity Quantification:
- Synonymy: ~54% of verbs span multiple clusters (polysemy), and clusters contain an average of 1.59–1.97 verbs, confirming the prevalence of synonyms.
- Multi-perspectives: Over 70% of images belong to more than one cluster, indicating that a single image often supports multiple valid descriptive perspectives.
Metric Performance: K-Means clustering using Llama-3.2-90B embeddings yielded the best balance of internal cohesion (Silhouette, Calinski-Harabasz) and external validity (WordNet purity).

Model Evaluation Improvements

Evaluating various supervised models (ResNet-50, CLIP, Llama-11B) and zero-shot LLMs (GPT-4o mini, Llama-3.2-90B) revealed significant performance gains under the cluster-based metric:

Accuracy Gains: All models showed substantial Top-1 accuracy improvements. For instance, the Llama-11B supervised model improved from 56.0% (Exact Match) to 72.1% (Cluster-based). Zero-shot LLMs showed even larger relative gains (e.g., GPT-4o mini closed-prompt improved from 17.3% to 39.2%).
Source of Improvement: The breakdown analysis indicates that multi-perspective understanding accounts for the majority of the gain (8–27% improvement), while synonym resolution contributes a smaller but significant portion (3–6%).
Human Alignment: The cluster-based evaluation demonstrated the highest agreement with human judgments, outperforming both exact-match and VLM-as-a-judge baselines in terms of Kendall's $\tau$ (ranking agreement) and Krippendorff's $\alpha$ (label agreement). Unlike VLM-as-a-judge, which showed high instability across prompt variations, the cluster-based method provided consistent, robust alignment.

Limitations of Alternative Metrics

The paper demonstrates that CLIPScore is ill-suited for verb-centric evaluation. Even when using full captions, CLIPScore is dominated by contextual words, rendering changes in the verb negligible. Verb-only CLIPScore achieved only ~25% Top-1 accuracy, failing to reliably retrieve human-validated labels.

Significance and Contributions

The paper claims three primary contributions:

Novel Framework: The proposal of a vision-language clustering framework that constructs sense groups based on images and activity verbs, validated by internal and external metrics.
Superior Evaluation: Demonstration that cluster-based evaluation offers a more accurate and nuanced assessment of visual activity recognition systems compared to traditional exact-match accuracy, better reflecting the models' true semantic capabilities.
Ambiguity Quantification: A quantitative analysis of the imSitu dataset revealing the extent of synonymy and multi-perspective ambiguity, showing that standard evaluation metrics fail to capture these inherent complexities.

The authors conclude that while their experiments focus on imSitu, the framework is designed to be adaptable to other visual activity datasets. They acknowledge limitations, including reliance on LLM-generated verbs rather than exhaustive human annotation and the computational cost of the offline clustering process, but argue that the resulting fixed set of targets amortizes this cost for future evaluations.