💻 computer science

CLARA: Clip-Level Multimodal Alignment with VLM-Derived Rationales for Hateful Video Detection

This paper proposes CLARA, a novel clip-level multimodal framework that enhances hateful video detection by modeling videos as fine-grained sequences with adaptive alignment, contrastive learning for temporal dependencies, and VLM-derived rationales, achieving state-of-the-art performance across multiple datasets.

Original authors: Yuchen Zhang, Shuang Dai, Zeyu Fu, Yunfei Long, Ravi Shekhar, Haralambos Mouratidis

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuchen Zhang, Shuang Dai, Zeyu Fu, Yunfei Long, Ravi Shekhar, Haralambos Mouratidis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

On the screens of billions of people, a new kind of conversation is taking place. It happens in the rapid, scrolling streams of video platforms where a joke, a news clip, or a personal vlog can shift in an instant from harmless to harmful. Unlike a written post, where the meaning is fixed in words, a video carries a complex layering of sound, moving images, and spoken language. The danger lies in how these elements interact. A hateful message might not be shouted in a single sentence; instead, it can emerge slowly, building up through a sequence of frames, a change in tone, or a visual cue that only makes sense when viewed in the context of what came before. Detecting this kind of content is a massive challenge for the internet, because the most dangerous signals are often brief, hidden, and dependent on the specific moment they appear.

For years, computer scientists have tried to teach machines to spot this kind of harm. Early efforts focused on text, looking for specific angry words. Later, they moved to images and memes, trying to understand how pictures and captions work together. But videos present a unique problem. They are not just a collection of images or a transcript of speech; they are a flowing timeline where meaning changes over time. If a computer looks at a whole video as a single block, it often misses the small, critical moments where the hate actually happens. It is like trying to understand a story by reading only the first and last sentence; the crucial plot points in the middle get lost in the summary.

A team of researchers at the University of Essex and the University of Exeter has proposed a new way to solve this. They call their system CLARA. Instead of treating a video as one big, unchangeable object, CLARA breaks it down into smaller, meaningful pieces. Imagine a video as a long sentence; CLARA does not read the whole thing at once. Instead, it pauses at every natural break in the speech, creating a series of short clips that align with what is being said. By focusing on these small segments, the system can catch the fleeting moments where a hateful idea is introduced, even if that idea is buried in a minute of neutral content.

The researchers found that hateful videos often rely on a mix of clues. A speaker might use a calm voice while showing a disturbing image, or use a neutral phrase while the background music turns aggressive. To handle this, CLARA uses a flexible system that decides, for each short clip, which clues matter most. Sometimes the text is the key; other times, it is the sound or the visual scene. The system does not force a single rule for the entire video. Instead, it adapts, weighing the different types of information differently as the video progresses. This allows it to see how a hateful message evolves, rather than just looking for a static pattern.

To make sure the system understands the bigger picture, the researchers added a second layer of intelligence. They used a powerful artificial intelligence tool, trained to describe and reason about images and text, to generate a high-level summary of the video's intent. This summary acts as a guide, helping the system understand how the small clips fit together to form a larger, potentially harmful narrative. The system then combines these detailed clip-level observations with the high-level summary, creating a complete picture of the video's content. This approach ensures that the system does not just see isolated moments, but understands the flow of the argument or the story being told.

The team tested this new method on three different collections of hateful videos, drawn from various social media platforms around the world. The results were clear: the new system consistently outperformed the best existing methods. It was better at correctly identifying hateful videos without mistakenly flagging harmless ones. The researchers also ran tests to see what would happen if they removed parts of the system. When they took away the ability to break videos into clips, or when they removed the high-level guidance, the system's performance dropped. This confirmed that every part of their design was necessary. The ability to focus on small, specific moments while keeping an eye on the overall context was the key to success.

The study suggests that the future of detecting harmful content lies in this kind of detailed, time-aware analysis. By respecting the way humans actually experience video—piece by piece, moment by moment—machines can finally learn to see the subtle, evolving nature of hate speech. The researchers did not claim to have solved the problem of online hate forever, but they have provided a significantly sharper tool for the job. Their work shows that to understand the danger in a video, you cannot just look at the whole picture; you must watch the story unfold, one small clip at a time.

Technical Summary: CLARA – Clip-Level Multimodal Alignment with VLM-Derived Rationales for Hateful Video Detection

Problem Statement

Hateful video detection is a critical yet underexplored challenge in content moderation, particularly as video-centric social media platforms grow. Unlike text or static multimodal content (e.g., memes), hateful meaning in videos often arises from complex, temporally dependent interactions between speech, audio, and visual cues. These signals are frequently brief, implicit, and context-sensitive, emerging progressively across multiple clips rather than being contained within a single frame or utterance.

Existing approaches predominantly rely on coarse video-level modeling, where entire videos are encoded into global representations before fusion and classification. This paradigm suffers from three key limitations:

Loss of Temporal Granularity: Compressing a video into a single representation risks diluting short, decisive hateful moments with long neutral segments.
Rigid Fusion Strategies: Fixed fusion rules fail to adapt to dynamic shifts in modality contributions (e.g., hate shifting from spoken words to visual context) over time.
Disconnected Local-Global Modeling: Prior work rarely explicitly models the relationship between local cues and the broader discourse context, which is essential for interpreting implicit hate.

Methodology: The CLARA Framework

The authors propose CLARA (Clip-Level Multimodal Alignment with VLM-Derived Rationales), a framework that shifts the basic modeling unit from the whole video to utterance-aligned clips. The architecture consists of six main components:

1. Utterance-Aligned Clip Segmentation

Instead of fixed-length segmentation, videos are segmented based on utterance boundaries using Whisper-large-v3 transcriptions and timestamps. This ensures each clip corresponds to a semantically coherent unit. Silent intervals are handled by merging pauses shorter than 1 second into adjacent clips while retaining longer pauses as standalone segments to preserve discourse structure.

2. MoE-Based Clip Encoder

For each clip, multimodal features are extracted:

Audio: Raw embeddings from Whisper-large-v3.
Visual: Frame-level embeddings from a pre-trained ViT, with frame budgets allocated proportionally to clip duration.
Text: Concatenated embeddings from transcriptions and OCR text (extracted via PaddleOCR) encoded by BERT.

These features are projected into a shared latent space and fused using a Mixture-of-Experts (MoE) encoder. A gating network dynamically routes inputs to a subset of experts (Top-K), enabling input-dependent, adaptive multimodal alignment rather than static fusion. A load-balancing loss ensures efficient expert utilization.

3. Local-Global Segment Contrastive Learning

To bridge local cues and global context, a contrastive objective is introduced. The model constructs:

Local Segments: Short sequences of consecutive clips.
Global Segments: Longer temporal spans covering the video.
The objective enforces consistency between local and global segments from the same video (positive pairs) while pushing apart segments from different videos (negative pairs). This encourages the model to capture both short-term signals and long-range temporal dependencies.

4. VLM-Driven Rationale Generation

A Vision-Language Model (Qwen3VL-8B-Instruct) generates video-level rationales through a two-step prompting process:

Objective Analysis: Describing visuals, summarizing text, and explaining their relationship.
Hate Verification: Determining hatefulness (explicit/implicit) and providing evidence-based reasons.
These rationales serve as high-level semantic guidance.

5. Rationale-Gated Transformer

The generated rationale embedding and the sequence of clip embeddings are integrated via a gated Transformer. A lightweight gating mechanism modulates the contributions of the rationale and clip branches before temporal encoding. This allows the model to dynamically weigh high-level semantic reasoning against fine-grained clip evidence before aggregating them into a final video representation.

6. Training Objective

The model is optimized using a combination of:

Supervised Cross-Entropy Loss: For final hate classification.
Contrastive Loss ( $L_{CL}$ ): For local-global alignment.
Load-Balancing Loss ( $L_{LB}$ ): For MoE stability.

Key Contributions

Clip-Level Paradigm: CLARA moves beyond coarse video-level modeling by explicitly representing videos as sequences of utterance-aligned clips, preserving fine-grained temporal units where hateful cues emerge.
Adaptive Architecture: The framework combines a MoE clip encoder for dynamic fusion, a local-global contrastive objective for cross-scale temporal consistency, and a gated Transformer that integrates VLM-derived rationales for semantic guidance.
State-of-the-Art Performance: Extensive experiments demonstrate that CLARA consistently outperforms existing baselines across multiple datasets.

Experimental Results

The authors evaluated CLARA on three datasets: HateMM, MultiHateClip (Chinese and English subsets), and DeHate.

Performance: CLARA achieved state-of-the-art results across all datasets and metrics (Accuracy, Macro-F1, Macro-Precision, Macro-Recall).
- On HateMM, CLARA achieved 87.9% Accuracy and 87.2% Macro-F1, outperforming the second-best method (MM-HSD) by 4.0% in Accuracy (0.879 vs 0.839) and 3.9% in Macro-F1 (0.872 vs 0.833).
- Similar significant improvements were observed on MHC_CN, MHC_EN, and DeHate, with absolute improvements across all datasets ranging from 2.4% to 5.3% in Accuracy and 2.7% to 7.4% in Macro-F1.
Ablation Studies: Removing any component (modalities, MoE, contrastive learning, rationales, or the gated Transformer) resulted in performance degradation. Notably, removing the gated Transformer caused a significant drop, highlighting the importance of explicit temporal modeling.
Parameter Analysis:
- MoE: Optimal performance was found with 10 experts and Top-2 routing, though configurations with 8–10 experts generally produced strong results.
- Contrastive Weight: A moderate weight ( $\lambda_{CL} \approx 0.4$ ) provided the best trade-off between alignment and classification.
- Frame Budget: A relatively small budget (40 frames) was sufficient, with larger budgets offering no additional benefit.

Significance and Claims

The paper claims that effective hateful video detection requires not just the integration of multiple modalities, but when and at what temporal granularity they are aligned. By shifting multimodal interaction to the utterance-aligned clip level, CLARA mitigates the dilution of decisive hateful cues found in global video modeling.

The authors assert that their approach effectively addresses the challenge of implicit and context-dependent hate by:

Preserving fine-grained temporal structure.
Explicitly linking local evidence with broader discourse via contrastive learning.
Leveraging VLM-derived rationales to guide interpretation at the sequence level rather than post-compression.

The work positions CLARA as a robust, state-of-the-art solution that advances the field beyond fixed fusion strategies and coarse-grained representations, offering a more nuanced understanding of how hateful meaning unfolds over time.