🤖 machine learning

An Evaluation of Hybrid Annotation Workflows on High-Ambiguity Spatiotemporal Video Footage

This paper demonstrates that integrating tuned encoder pre-annotations into a human-in-the-loop workflow significantly reduces manual annotation time for high-ambiguity spatiotemporal video footage by 35% for most users, while establishing a rigorous framework to evaluate the trade-offs between algorithmic speed and human verification integrity.

Original authors: Juan Gutiérrez, Victor Gutiérrez, Ángel Mora, Silvia Rodriguez, José Luis Blanco

Published 2026-02-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Juan Gutiérrez, Victor Gutiérrez, Ángel Mora, Silvia Rodriguez, José Luis Blanco

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Labeling Videos is Exhausting

Imagine you have a massive library of security camera footage. Your job is to watch every single second and draw a box around every time something "weird" or "dangerous" happens (like a fight, a fall, or a theft). You also have to write down exactly when it started and when it ended.

Doing this manually is like trying to paint a masterpiece by hand, one tiny dot at a time. It takes forever, it's boring, and different people will draw the boxes in slightly different places. This is the "gold standard" for training AI, but it's too slow and expensive to do for huge amounts of video.

The Proposed Solution: The "Smart Assistant"

The researchers asked: What if we give the human annotator a "smart assistant" that does the heavy lifting first?

Instead of starting with a blank screen, the computer runs a special AI model (a "Vision-Language Model") that scans the video and says, "Hey, I think this 10-second chunk looks like a theft, and this next chunk looks normal." These are called Pre-Annotations.

The human's job then changes from "Creator" (drawing everything from scratch) to "Editor" (checking the AI's work and fixing mistakes). It's like the difference between writing a novel from scratch versus editing a draft written by a ghostwriter.

The Experiment: A Controlled Test

To see if this "Smart Assistant" actually helps without tricking people, the researchers ran a strict experiment:

The Players: 18 volunteers (mostly university students).
The Task: They had to label 30 different videos.
The Twist: Each person did two types of tasks:
1. The Hard Way: Labeling 5 videos with no help (just raw footage).
2. The Easy Way: Labeling 5 videos where the AI had already drawn the rough outlines.
The Setup: They swapped the order so that no one got an unfair advantage from just "getting used to the tool."

The Results: Faster, But Did They Lose Their Minds?

The researchers were worried about a specific trap: The "Yes-Man" Effect.
If you give someone a draft, they might just say "Okay" to everything the computer wrote, even if the computer is wrong. They might agree with the AI just to finish faster, losing their own judgment. This is called "semantic drift."

Here is what they found:

1. Speed: The "Time Machine" Effect

Result: The "Smart Assistant" made people 35% faster on average.
Analogy: Imagine you are packing a suitcase. Doing it alone takes 20 minutes. If someone else packs 70% of it for you, and you just have to zip it up and fix a few socks, you finish in 13 minutes.
Detail: 72% of the volunteers were faster with the AI help. The more they used the tool, the better they got at quickly verifying the AI's suggestions.

2. Quality: Did They Just Agree with the AI?

Result: Surprisingly, no. The people who used the AI didn't just blindly copy the computer.
Analogy: Imagine a group of friends trying to decide where a movie scene ends. Without help, everyone guesses differently (some say it ends at 1:00, others at 1:05). With the AI, the AI says "It ends at 1:02." The friends still argue a bit, but they all agree much more closely on the 1:02 mark.
The Science: The researchers measured how much the volunteers agreed with each other. The group using the AI agreed with each other more than the group working alone. This means the AI acted as a "ruler" or a "standard," helping everyone draw their lines in the same place without forcing them to accept wrong answers.

3. The "Jitter" Problem

Result: Without help, human labels were "jittery" (wobbly and inconsistent). With help, the labels became "smooth" and consistent.
Analogy: Think of drawing a line on a piece of paper. If you do it freehand, your hand shakes a little (jitter). If you use a ruler (the AI), the line is straight. The paper claims the AI provided the "ruler" that made the humans more consistent, without changing what they were drawing.

The Takeaway

This paper proves that giving humans a "first draft" from an AI is a win-win for video labeling:

It saves time: People finish the job much faster.
It keeps quality high: People don't just blindly agree with the AI; they use it as a guide to be more consistent with each other.
It's safe: The AI didn't trick the humans into making bad decisions; it just helped them stop shaking their hands while drawing.

The researchers also released their code and the data from the volunteers' mouse clicks and typing so other scientists can check their work and try it themselves. They emphasized that while this helps, humans must still be in charge to catch the AI when it makes mistakes, especially when the video content is sensitive or violent.

Technical Summary: An Evaluation of Hybrid Annotation Workflows on High-Ambiguity Spatiotemporal Video Footage

Problem Statement
Manual annotation remains the gold standard for high-quality, dense temporal video datasets but is inherently time-consuming, expensive, and variable between annotators. This bottleneck is particularly severe in high-ambiguity video footage (e.g., open-world surveillance), where events lack clear boundaries, involve occlusions, and are subject to cultural and contextual interpretation. While Vision-Language Models (VLMs) and Human-in-the-Loop (HITL) workflows have been adopted to shift the annotator's role from "create" to "verify," there is a lack of rigorous, controlled studies quantifying how AI suggestions affect annotation efficiency, inter-annotator consistency, and the semantic integrity of labels. Specifically, there is a risk that AI assistance could induce "semantic drift," where human labeling shifts toward the model's priors, potentially reducing label validity even as agreement increases.

Methodology
The authors present a controlled, counterbalanced A/B study involving 18 volunteer annotators labeling 30 videos from open-world datasets (UCF-Crime, UCA, XD-Violence, ActivityNet Captions). The study evaluates a hybrid HITL workflow designed for dense temporal segmentation.

Pre-Annotation Pipeline: The system utilizes a resource-efficient cross-modal pipeline based on a fine-tuned CLIP-style encoder. Video frames are encoded, and temporal pooling aggregates them into video-level embeddings. These are aligned with text embeddings via contrastive learning (optimizing symmetric Cross-Entropy loss).
Hierarchical Clustering: To generate "strong clusters" (Pre-Annotations), the pipeline applies recursive spherical k-means clustering to the feature space. This creates a hierarchical tree structure, stopping subdivision when a semantic cohesion threshold (cosine similarity $\tau=0.85$ ) is met or a maximum depth is reached. These clusters serve as initial, read-only suggestions.
User Interface: The workflow is integrated into Label Studio. In the "Assisted" condition, annotators view pre-populated segments on a temporal timeline and must explicitly duplicate, adjust boundaries, and correct class labels. In the "Unassisted" condition, annotators create segments from scratch.
Evaluation Framework: The study measures three dimensions:
1. Efficiency: Task duration and interaction telemetry.
2. Consistency: Inter-annotator agreement (Homogeneity) and alignment with a Consensus Ground Truth (derived from a 6-vote overlap strategy).
3. Standardization: Structural coherence measured via Silhouette Scores in the latent space to detect stochastic variance ("jitter").

Key Contributions

Controlled Stress Test: A study isolating the effect of Pre-Annotations on human behavior in high-ambiguity settings, moving beyond simulated studies to real-world open-world footage.
Reproducible Hybrid Workflow: An end-to-end, lightweight pipeline (Pre-Annotation + UI) released as open-source code, enabling future benchmarking of human-model interaction.
Multi-Axis Evaluation Protocol: A framework that goes beyond speed to measure agreement, consensus, and latent-space semantic validity to detect potential semantic drift.
Public Dataset: Release of anonymized interaction traces (time, edits, label changes) to support research on human-centric dataset construction.

Results

Efficiency: The hybrid workflow reduced annotation time by a median of 35% for 72% of participants. The mean time reduction across the cohort was 23.11%. Efficiency gains were robust across varying video durations and class complexities, and the benefit increased as users became familiar with the tool.
Consistency (Agreement): Contrary to fears of passive anchoring, the assisted group maintained alignment with the Consensus Ground Truth (Adjusted Mutual Information $\approx 0.64$ ) identical to the unassisted group. This indicates that Pre-Annotations did not distort user judgment or induce deviation from collective semantic consensus.
Homogeneity (Stability): The assisted group demonstrated significantly higher inter-annotator agreement (AMI: 0.67) compared to the unassisted group (AMI: 0.62). The Pre-Annotations acted as a "semantic anchor," reducing the stochastic noise and boundary "jitter" inherent in human labeling without enforcing an arbitrary correct answer.
Standardization: The Silhouette Score, a proxy for internal structural consistency, improved from 0.28 (unassisted) to 0.41 (assisted), a 52.10% relative improvement. This confirms the workflow acts as a regularizer, filtering out inter-annotator noise while preserving the underlying semantic consensus.

Significance and Claims
The paper claims that model-generated Pre-Annotations can substantially improve the efficiency of dense temporal video segmentation without sacrificing label validity. The authors argue that the workflow successfully reframes annotation as a verification-and-editing task, reducing cognitive load and manual effort. Crucially, they demonstrate that these efficiency gains are not achieved by forcing annotators to blindly accept model outputs; rather, the system provides a structural scaffold that helps annotators converge on clearer temporal boundaries. The work provides a validated methodology for studying human-AI interaction dynamics and offers a practical foundation for scaling the creation of complex, densely labeled video datasets, provided that ethical considerations regarding annotator well-being and agency are maintained.

The Big Problem: Labeling Videos is Exhausting

The Proposed Solution: The "Smart Assistant"

The Experiment: A Controlled Test

The Results: Faster, But Did They Lose Their Minds?

The Takeaway

More like this