💻 computer science

CEZSAR: A Contrastive Embedding Method for Zero-Shot Action Recognition

The paper proposes CEZSAR, a novel contrastive learning method for Zero-Shot Action Recognition that addresses semantic gaps and domain shifts by aligning video and text embeddings in a joint space using an automatic negative sampling procedure, achieving state-of-the-art results on UCF-101 and Kinetics-400 datasets.

Original authors: Valter Estevam, Rayson Laroca, Helio Pedrini, David Menotti

Published 2026-05-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Valter Estevam, Rayson Laroca, Helio Pedrini, David Menotti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to recognize human actions, like "riding a horse" or "playing basketball." Usually, to teach a robot, you need to show it thousands of videos of people doing those specific things and label them one by one. But what if you want the robot to recognize a new action, like "juggling chainsaws," that it has never seen before? You can't show it examples because they don't exist in its training data. This is the challenge of Zero-Shot Action Recognition.

The paper introduces a new method called CEZSAR to solve this. Here is how it works, explained simply:

The Two Big Problems

The authors say that trying to teach a robot new actions without showing examples is hard because of two main "bumps in the road":

The Semantic Gap (The Language Barrier):
Imagine you have a robot that speaks "Visual" (it sees pixels and shapes) and another that speaks "Text" (it understands words). If you tell the text-robot, "This is a horse race," it knows the concept. But the visual-robot sees a blurry image of a horse and a track. The problem is that the "idea" of a horse race in the text world doesn't perfectly match the "picture" of a horse race in the visual world. They are like two people speaking different dialects; they understand the same thing, but the details get lost in translation.
- The Paper's Fix: CEZSAR builds a universal translator. It forces the visual robot and the text robot to learn a shared language where the picture of a horse race and the sentence "horse race" sit right next to each other in their memory.
The Domain Shift (The Context Trap):
Imagine you train a robot only on videos of people running in a park. If you then ask it to recognize someone running on a treadmill in a gym, it might get confused because the background (the "domain") is totally different. The robot memorized the park, not the act of running.
- The Paper's Fix: The authors realized that while the visual world changes a lot (different parks, different gyms), the text description of "running" stays the same. By anchoring the visual learning to the text description, the robot learns to ignore the confusing background and focus on the action itself.

How CEZSAR Works (The Recipe)

The method uses a "contrastive" approach, which is like a game of "Hot and Cold."

The Setup: The system takes a video and a sentence describing it.
The Goal: It tries to make the video and its matching sentence "hug" each other in a mathematical space (getting very close).
The Twist (Hard Negative Sampling): This is the clever part. The system needs to learn what doesn't match. Instead of humans manually finding bad matches, the computer automatically generates them.
- It takes a video of someone "riding a horse."
- It grabs a sentence about "riding a horse" (the good match).
- It grabs a sentence about "riding a bike" or "cooking pasta" (the bad match).
- It forces the computer to push the "cooking" sentence far away from the "horse" video.
- The Magic: They use a smart trick to find these "bad matches" automatically without human help, creating millions of training examples in just a few hours on a single computer.

The Results

The authors tested this on two famous video datasets (UCF-101 and Kinetics-400).

The Score: Their method got significantly higher accuracy than previous methods. For example, on a test with 101 different actions it had never seen, it improved accuracy by about 10 percentage points compared to the next best method.
Why it worked: By using the text descriptions to guide the visual learning, the robot became much better at distinguishing between similar-looking actions (like "horse riding" vs. "horse racing") and wasn't confused by different backgrounds.

In a Nutshell

CEZSAR is a new way to teach computers to recognize actions they've never seen before. Instead of just memorizing pictures, it teaches the computer to link pictures with their descriptions, using a smart game of "match and mismatch" to bridge the gap between what we see and what we read. This allows the computer to understand new actions quickly and accurately, even if it has never watched a video of them before.

Technical Summary: CEZSAR

Problem Definition

The paper addresses Zero-Shot Action Recognition (ZSAR) in videos, a task aimed at classifying action instances belonging to classes that were not present during the model's training phase. The authors identify two persistent challenges in existing ZSAR approaches:

The Semantic Gap: A misalignment between the visual domain (represented by CNNs or Transformers) and the textual domain (represented by language models). Even in joint embedding spaces, the distribution of visual features often diverges from their semantic counterparts. For instance, actions like "Pommel Horse" and "Balance Beam" share similar scene structures (gymnasiums) but differ in specific motions and equipment, leading to confusion in embedding spaces.
Domain Shift: The discrepancy in probability distributions between the training set and the unseen test set. Since the test set is unknown during training, models struggle to generalize to new data distributions.

Existing methods often rely on human-defined attributes or simple word embeddings (e.g., Word2Vec, GloVe), which fail to capture fine-grained semantic differences or require extensive manual effort. Furthermore, while large-scale pre-trained models like CLIP show promise, they require massive computational resources and data (hundreds of millions of pairs) that are often inaccessible.

Methodology: CEZSAR

The authors propose CEZSAR (Contrastive Embedding Method for Zero-Shot Action Recognition), a model designed to learn a joint embedding space for videos and natural language descriptions using contrastive learning.

Architecture

The model consists of two primary modules:

Visual Embedding Module (VEM):
- Videos are sampled at 1 frame per second (FPS).
- Frames are encoded using a pre-trained Convolutional Neural Network (CNN), specifically ResNet-152, producing a feature stack.
- A fully connected layer reduces dimensionality, followed by a Transformer Encoder with multi-head self-attention to model temporal information.
- The output is projected into a joint semantic space (128 dimensions).
Sentence Embedding Module (SEM):
- Natural language descriptions are processed using Sentence-BERT (SBERT) to generate 768-dimensional embeddings.
- These are projected via a fully connected layer into the same joint semantic space.

Training Strategy: Hard Negative Sampling

A core innovation is the automatic hard negative sampling procedure, which eliminates the need for human-annotated negative pairs:

Semantic Filtering: The model uses a pre-trained SBERT model to evaluate the similarity between sentences. Sentences with high cosine similarity (above a threshold $\tau$ ) are considered semantically similar.
Object-Based Augmentation: The system detects objects in the video and generates descriptive sentences for them. It then selects negative descriptions that are sufficiently different from both the human-annotated description and the object-generated descriptions.
Triplet Generation: This process generates triplets consisting of (video, positive description, negative description). The authors report generating approximately three million triplets from the ActivityNet Captions dataset.
Loss Function: The model is trained using a Triplet Loss function, which minimizes the distance between a video and its positive description while maximizing the distance to its negative description.

Classification

During inference, the model maps both the input video and the class prototypes (defined by sentence descriptions) into the joint embedding space. Classification is performed using a nearest neighbor rule based on cosine similarity. The final video embedding ($VidE$) can optionally incorporate object-level semantic information ( $O(v)$ ) alongside raw visual features ($VE(v)$), controlled by weighting parameters $\alpha$ and $\beta$ .

Key Contributions

The paper outlines three main contributions:

A Novel Cross-Modal Contrastive Method: CEZSAR effectively associates visual features with sentence descriptions, significantly reducing the semantic gap between modalities.
Modular Projection: The architecture allows videos and descriptions to be projected via distinct sub-networks, facilitating the inclusion of additional modalities (text, images, or other videos) without retraining the entire contrastive model.
Efficiency and Robustness: The method achieves state-of-the-art results on standard benchmarks using a single GPU and a fraction of the data and training time required by large-scale models like CLIP.

Experimental Results

The method was evaluated on the UCF-101 and Kinetics-400 datasets under various zero-shot split configurations (e.g., 0% seen classes, 100% unseen classes).

UCF-101: Under the 0/101 configuration (all classes unseen), CEZSAR achieved 95.2% accuracy (visualized in Fig. 1b), a significant improvement over the baseline ZSARCAP (67.2%) and other state-of-the-art methods. The inclusion of object-level semantics provided marginal but consistent improvements.
Kinetics-400: The model outperformed all previous state-of-the-art methods across all test configurations (25, 100, and 400 unseen classes). For the 400-class split, it achieved 23.8% accuracy, surpassing the previous best of 20.4%.
Hard Samples: On a subset of 15 difficult, highly similar classes (e.g., distinguishing between "horse riding" and "horse race"), CEZSAR achieved 63.3% accuracy compared to 47.4% for ZSARCAP. The confusion matrices indicate a substantial reduction in misclassification errors, particularly for groups involving similar equipment or scenarios.

Significance and Claims

The authors claim that CEZSAR demonstrates that contrastive learning is a straightforward yet effective approach for bridging the semantic gap in ZSAR. By conditioning the learning of visual features on textual descriptions—a modality less susceptible to domain shift—the model naturally mitigates the domain shift problem.

The paper emphasizes that high performance in zero-shot recognition does not necessarily require massive-scale pre-training infrastructure (e.g., thousands of GPUs and hundreds of millions of data pairs). Instead, a carefully designed contrastive framework with automatic negative sampling can achieve superior results on standard datasets using significantly fewer resources (1000x fewer visual representations and 100x fewer data pairs than CLIP-scale models, trained in a few hours on a single GPU). The work suggests that the primary driver of performance gains is the effective reduction of the semantic gap rather than merely the volume of semantic information.