💻 computer science

Detection of Hate and Threat in Digital Forensics: A Case-Driven Multimodal Approach

This paper introduces a case-driven multimodal framework for digital forensics that enhances hate and threat detection by dynamically selecting text analysis, multimodal fusion, or image-only reasoning based on the specific configuration of heterogeneous evidence, thereby ensuring forensic traceability and avoiding unjustified modality assumptions.

Original authors: Ponkoj Chandra Shill

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Ponkoj Chandra Shill

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a crime, but instead of finding a single, clear clue, you are handed a messy pile of evidence. Some clues are written on a piece of paper (text), some are scribbled inside a photograph (text inside an image), and some are just pictures with no words at all.

In the world of Digital Forensics, investigators often face this exact problem. They need to find evidence of hate speech, threats, or violence in a chaotic mix of screenshots, scanned documents, and chat logs.

The paper you provided introduces a new "smart detective system" designed to handle this mess without making dangerous assumptions. Here is how it works, explained simply:

1. The Problem: The "One-Size-Fits-All" Mistake

Most current computer programs that look for hate speech are like detectives who only know how to read clean, typed letters.

If you give them a clear text message, they are great.
If you give them a photo with text written on it (like a meme), they might get confused because the text is blurry or hard to read.
If you give them a photo with no text (just a picture of a weapon), they might miss the point entirely because they are waiting for words.

The old way was to force all evidence through the same filter, which often led to wrong conclusions or missed clues.

2. The Solution: The "Case-Driven" Detective

The new system proposed in this paper acts like a seasoned detective who first inspects the evidence before deciding how to analyze it. It doesn't just guess; it asks, "What exactly do I have here?"

It sorts every piece of evidence into one of four "Cases" (or scenarios):

Case 1: The "Meme" Scenario. You have a picture with text written inside it (like a screenshot of a chat).
- The Detective's Move: The system uses a special tool (called OCR) to "read" the blurry text inside the picture, but it knows this reading might be a bit shaky. It analyzes the picture and the shaky text separately, then combines them.
Case 2: The "Context" Scenario. You have a picture, but the text isn't in the picture. Instead, there is a police report or a chat log next to the picture explaining what it is.
- The Detective's Move: This text is usually very clear and reliable. The system treats this as a "high-quality" clue and gives it extra weight when making a decision.
Case 3: The "Silent Photo" Scenario. You have a picture with no text at all (e.g., a photo of a knife).
- The Detective's Move: Since there are no words to read, the system looks only at the visual clues. It doesn't try to invent a story; it just analyzes what it sees.
Case 4: The "Total Silence" Scenario. (This is when both text types are missing, similar to Case 3).
- The Detective's Move: Pure visual analysis.

3. The Magic Trick: The "Frozen Label" Menu

Imagine the detective has a fixed menu of 10 possible crimes (like "Threat of Violence," "Harassment," "Hate Speech," or "Nothing Bad"). This menu never changes.

Old Way: Different computers might invent their own labels, making it hard to compare results.
New Way: Whether the computer is reading text or looking at a picture, it must pick an answer from this same, fixed menu. This ensures that a "Threat" identified by a text reader is the exact same thing as a "Threat" identified by a picture reader.

4. The Final Verdict: The "Weighted Vote"

Once the system analyzes the evidence based on the case, it needs to make a final decision. It uses a voting system:

Visual Evidence: Gets 1 vote.
Text from a Police Report (Clean): Gets 1.2 votes (because it's very reliable).
Text from a Blurry Screenshot (OCR): Gets 1 vote (but the system knows it might be noisy).

If the evidence is missing (e.g., no text at all), that vote simply doesn't happen. The system doesn't force a vote; it just counts the votes that do exist. This prevents the system from guessing when it has no information.

Why Does This Matter?

Think of this system as a transparent, honest assistant for human investigators.

It doesn't hide its work: You can see exactly why it flagged something. Was it the picture? The text? Both?
It's safe: It won't make up facts. If there is no text, it won't pretend there is.
It's flexible: It works whether the evidence is a clean document, a messy screenshot, or just a photo.

The Bottom Line

This paper teaches us that in digital forensics, context is king. You can't treat a blurry screenshot the same way you treat a clean police report. By building a system that first checks what kind of evidence it has, and then chooses the right tool to analyze it, investigators can find threats and hate speech more accurately and with more confidence in a court of law.

It's the difference between a detective who blindly guesses and one who carefully examines the evidence before speaking.

1. Problem Statement

Digital forensic investigations increasingly rely on heterogeneous evidence, including images, screenshots, scanned documents, and narrative reports. These artifacts often contain explicit or implicit expressions of hate, threats, violence, or harassment. However, existing automated detection tools face significant limitations in forensic contexts:

Assumption of Clean Data: Most models assume clean, structured text input, whereas forensic text is often fragmented, noisy (e.g., OCR errors), or entirely absent.
Modality Isolation: Tools often treat image and text modalities in isolation or apply fixed multimodal fusion strategies that do not account for evidence availability.
Lack of Traceability: Standard "black-box" deep learning approaches fail to provide the evidentiary traceability and auditability required for legal defensibility.
Unjustified Inference: Applying a single analysis strategy to all scenarios can lead to unreliable conclusions, such as inferring context from images when no textual evidence exists.

The core challenge is not model expressiveness, but evidentiary legitimacy: semantic inference must be conditioned strictly on the availability and quality of the evidence to avoid unsupported assumptions.

2. Methodology

The paper proposes a Case-Driven Multimodal Forensic Analysis Pipeline that explicitly determines the presence and source of textual evidence before applying semantic analysis. The framework operates in a zero-shot setting (no task-specific training) to ensure adaptability to evolving threat categories.

A. Evidence Classification (The "Cases")

The system first categorizes evidence into three distinct forensic scenarios based on modality availability:

Case 1 (Image with Embedded Text): The image contains visual text (e.g., screenshots). Text is extracted via Optical Character Recognition (OCR), denoted as $T_{ocr}$ . This modality is treated as distinct and noisy.
Case 2 (Image with Associated Text): The image is accompanied by contextual text ( $T_{assoc}$ ) from forensic reports, metadata, or chat logs. This text is human-authored, cleaner, and linked via shared identifiers/timestamps.
Case 3 (Image-Only): No textual information is available. Analysis relies solely on visual semantics.

B. Derived Data Situations (DS)

Based on the cases, the framework routes the evidence to one of four data situations (DS1–DS4) to determine which models to execute:

DS1: Both $T_{ocr}$ and $T_{assoc}$ present.
DS2: Only $T_{ocr}$ present.
DS3: Only $T_{assoc}$ present.
DS4: No text present (Image-only).

C. Model Architecture & Label Space

Frozen Semantic Label Space: All models map outputs to a fixed set of 10 forensic-relevant categories (e.g., Threat of Violence, Harassment, Hate Speech, Weapon-Related Content). This ensures semantic consistency across modalities.
Image Analysis: Uses a pretrained Vision-Language Model (CLIP with ViT-L/14 backbone) to compute similarity scores between the image and natural language prompts representing the frozen labels.
Text Analysis: Uses a zero-shot text classification model (DeBERTa-v3-large) to score both OCR-derived text and associated text against the same frozen labels.

D. Score-Level Fusion

Instead of learned end-to-end fusion, the framework uses an explicit, auditable weighted averaging scheme:
$s^{(fused)}_{\ell_k} = \frac{\sum_{m \in M} w_m s^{(m)}_{\ell_k}}{\sum_{m \in M} w_m}$
Where:

$M$ is the set of available modalities (Image, OCR, Associated Text).
$w_m$ are fixed weights based on forensic reliability: $w_{img}=1.0$ , $w_{ocr}=1.0$ , $w_{assoc}=1.2$ (reflecting higher confidence in human-authored text).
Missing modalities are excluded from the sum rather than penalized, ensuring the system degrades gracefully.

3. Key Contributions

Case-Driven Framework: A novel pipeline that explicitly distinguishes between embedded text, associated context, and image-only evidence prior to semantic analysis, mirroring forensic decision-making.
Modality-Aware Routing & Fusion: A strategy grounded in forensic reliability rather than learned fusion. It treats OCR text and associated text as distinct modalities with different reliability profiles.
Transparent Zero-Shot Analysis: The system operates without task-specific training, using a frozen label space to ensure interpretability and auditability.
Empirical Validation: Demonstrates consistent performance across heterogeneous evidence configurations, showing that multimodal fusion meaningfully refines decisions when text is available while maintaining robustness when it is not.

4. Experimental Results

The framework was evaluated on forensic-style evidence without fine-tuning.

Accuracy by Data Situation:
- DS3 (Associated Text Only): Highest accuracy (98.50%), validating the high reliability of human-authored context.
- DS4 (Image Only): 96.36% accuracy, showing strong visual semantic reasoning without linguistic cues.
- DS1 (Embedded + Associated): 95.45% accuracy.
- DS2 (Embedded Text Only): 94.29% accuracy. The lower performance highlights the impact of OCR noise but confirms the system's ability to utilize imperfect data.
Impact of Fusion:
- Multimodal fusion altered 34.29% of decisions in DS2 (OCR only) and 22.73% in DS1, indicating that text often corrects ambiguous visual predictions.
- In DS4, no changes occurred (as expected), confirming the system does not hallucinate context.
Interpretability: The system retains per-modality scores, allowing investigators to trace exactly why a decision was made (e.g., whether a threat was detected via visual objects or OCR text).

5. Significance and Implications

Forensic Defensibility: By conditioning inference on evidence availability and avoiding unjustified modality assumptions, the framework aligns with legal standards for evidence admissibility and auditability.
Practical Deployment: The zero-shot nature allows the system to adapt to new threat categories without retraining, a critical feature for evolving digital crime landscapes.
Decision Support: The paper positions the tool as a structured triage mechanism that supports human analysts rather than replacing them, providing transparent justification for automated assessments.
Handling Noise: The explicit separation of noisy OCR text from clean associated text prevents the degradation of high-confidence evidence by low-confidence data.

In conclusion, this research bridges the gap between advanced multimodal AI and the rigorous requirements of digital forensics, offering a robust, interpretable, and evidence-aware approach to detecting hate and threats in complex digital artifacts.