Technical Summary: Scribby – A Multi-Level LLM Framework for Semantic Video Analysis

Problem Statement

The exponential growth of video content across educational platforms, streaming services, and live streams has rendered manual analysis infeasible. While existing AI-driven video summarization tools exist, they typically operate at a coarse granularity, producing high-level overviews or extracting key frames. These approaches fail to capture the fine-grained semantic structure, thematic progression, and topical relationships necessary for comprehensive video analysis. Users seeking specific information within long-form content often lack efficient navigation tools, relying on predefined chapter markers (if available) or time-consuming manual scrubbing. Furthermore, current methods struggle to maintain robustness across both structured content (e.g., lectures) and unstructured content (e.g., livestreams) while preserving semantic accuracy at the micro level.

Methodology

Scribby is a multi-level framework designed to balance macro-level comprehension with micro-level semantic analysis. The system processes video content through three primary stages:

1. Video Transcription and Macro Summarization

• Transcription: Video audio is extracted via FFmpeg and transcribed using OpenAI's Whisper model, generating full transcripts with sentence-level timestamps.
• Macro Context: The complete transcript is summarized by a Large Language Model (LLM) to generate a high-level video overview. This macro summary serves as global context for downstream analysis and helps the LLM identify logical divisions to propose candidate chapter boundaries.

2. Sentence-Level Analysis and Semantic Clustering

Unlike methods that analyze sentences in isolation, Scribby constructs a multi-source context for each transcript sentence, including:

• The sentence itself.
• The macro-level video summary.
• Summaries of adjacent sentences (preceding and following).
• Historical context from previously analyzed sentences within the current semantic cluster.

LLM as a Judge: The system employs an LLM not merely as a summarizer but as a semantic judge. For each sentence, the LLM is prompted with the current verse being assembled, the sentence under evaluation, the global context, and explicit grouping criteria. The LLM determines whether the sentence continues the current semantic cluster or initiates a new "verse" (a semantically coherent cluster). This approach captures nuanced thematic shifts that pure embedding-based distance metrics might miss.

3. Vector-Space Representation and Visualization

• Embedding: Finalized verse summaries are embedded using Nomic-embed-text, a lightweight model optimized for semantic search (768 dimensions).
• Interactive Timeline: Verses are visualized on an interactive timeline. When a user submits a query, the system computes the cosine similarity between the query embedding and verse embeddings.
• Heatmap Visualization: Similarity scores are mapped to color gradients, creating a relevance heatmap across the video timeline. This allows users to visually identify thematic shifts and locate relevant sections rapidly.
• Retrieval: High-similarity verses can be combined to generate query-specific summaries, enabling content-aware extraction analogous to Retrieval-Augmented Generation (RAG) systems.

Key Contributions

1. Multi-Level Semantic Framework: Scribby integrates macro-level video comprehension with sentence-level semantic analysis, creating fine-grained, semantically structured representations ("verses") rather than single summaries.
2. Context-Aware LLM Judgment: The framework leverages LLMs as semantic judges, utilizing global transcript context and local sentence relationships to determine cluster boundaries, improving upon purely structural or distance-based segmentation.
3. Interactive Relevance Heatmaps: The system visualizes semantic structure via a color-coded timeline, enabling rapid identification of relevant sections based on user queries.
4. Structure-Agnostic Operation: The framework is designed to perform consistently across both well-structured educational content and unstructured, exploratory livestreams.

Experimental Results

The framework was evaluated using three experiments:

Experiment 1: Query Recall and Semantic Coherence

Testing on a tutorial video ("How to Create a Neural Network") showed that relevant queries (e.g., "Neural network") achieved higher average embedding similarity (0.526) and greater variance (0.106) compared to unrelated queries (e.g., "Marathon training," 0.339 avg, 0.028 std). This supports the hypothesis that relevant topics generate diverse similarity scores across contexts, while irrelevant topics produce consistently low scores.

Experiment 2: Alignment with Human-Defined Chapters

Scribby's verses were compared against 21 human-defined YouTube chapters from the same tutorial.

• Temporal Accuracy: The mean time offset between Scribby's verse boundaries and human chapters was 11.29 seconds, with most offsets under 10 seconds.
• Semantic Correspondence: In cases where LLM-generated summaries contained verbatim references to human chapter names, the mean cosine similarity between the verse summary and chapter name was 0.654, significantly higher than the baseline average of 0.428.
• Granularity: Scribby generated 51 verses compared to 21 human chapters, capturing fine-grained semantic transitions while maintaining alignment with high-level structural divisions.

Experiment 3: Performance Across Content Types

The system was tested on a structured video (planned development log) and an unstructured video (unedited livestream).

• Consistency: Average verse duration differed by only 8.6% (58s vs. 63s), indicating segmentation behavior is independent of content organization.
• Efficiency: Scribby processed videos in approximately 17–21% of the source video duration. In contrast, manual expert review typically requires 100% of the source duration. This represents a 5–6x efficiency gain over manual review.
• Computational Cost: Sentence-level semantic analysis dominated the computational cost (approx. 70% of processing time), while transcription and macro summary generation were minor components.

Significance and Limitations

Significance:
The paper establishes Scribby as a foundation for visualizing video analysis tools, demonstrating that LLM-based semantic judgment combined with vector-space retrieval can effectively balance computational efficiency with semantic richness. It advances beyond isolated solutions toward comprehensive video understanding by enabling query-based highlighting, semantic summarization, and interactive exploration. The framework proves particularly valuable for navigating long-form educational and technical content where thematic progression is critical.

Limitations:

• Embedding Limitations: Cosine similarity may not capture complete semantic equivalence; summaries with high similarity might share vocabulary but differ in intent, particularly with domain-specific terminology.
• LLM Dependency: Performance relies on LLM capabilities. Variations in prompt formulation, model state, or inherent stochasticity can lead to inconsistent segmentation decisions.
• Evaluation Scope: Current evaluation focused primarily on technical tutorials with clear structures. Generalizability to diverse content types (news, entertainment, creative works) requires further testing.
• Computational Overhead: Sentence-level analysis consumes 50–60% of total processing time due to repeated LLM inference, making real-time processing challenging without optimization (e.g., batching or model distillation).
• User Burden: Users must formulate effective queries to leverage the heatmap visualization, though the system currently lacks automatic query suggestion.

Future Directions:
The authors propose extending evaluation to diverse content types, incorporating multi-modal analysis (visual scene detection, OCR), enabling user-guided LLM prompting for domain-specific refinement, conducting formal qualitative user studies, and optimizing the pipeline for real-time processing and scalability.