💬 NLP

TR-EduVSum: A Turkish-Focused Dataset and Consensus Framework for Educational Video Summarization

This paper introduces TR-EduVSum, a new Turkish educational video dataset with 3281 human summaries, and proposes the AutoMUP framework to automatically generate high-quality, consensus-based gold-standard summaries by clustering and weighting meaning units, achieving semantic performance comparable to advanced large language models.

Original authors: Figen Eğin, Aytuğ Onan

Published 2026-04-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Figen Eğin, Aytuğ Onan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn a complex subject, like "Data Structures and Algorithms," from a 20-minute lecture video. The problem? The video is long, the speaker talks fast, and you don't have time to watch the whole thing. You need a summary. But who should write it? If you ask one person, they might miss the most important parts. If you ask ten people, you get ten different versions. Which one is the "truth"?

This paper, TR-EduVSum, solves that problem by creating a "Gold Standard" summary for Turkish educational videos, not by asking a computer to guess, but by asking a crowd of humans and finding the common ground.

Here is the breakdown of how they did it, using some everyday analogies:

1. The Problem: Too Many Voices, Too Little Time

Educational videos are great, but they are long. Existing AI tools (like the smart chatbots you know) are getting better at summarizing, but they can sometimes "hallucinate" (make things up) or get confused by the specific way Turkish people speak and structure their sentences. Also, there wasn't a good "answer key" to test if these AI tools were actually doing a good job.

2. The Dataset: The "Crowdsourced" Library

The researchers built a new library called TR-EduVSum.

The Content: They took 82 Turkish lecture videos about computer science.
The Crowd: They asked 138 different students to watch these videos and write their own summaries.
The Result: For every single video, they ended up with between 36 and 53 different summaries. That's over 3,200 human-written summaries in total!

Think of this like asking 50 different chefs to describe the taste of a specific soup. Some might focus on the salt, others on the herbs, and some on the texture.

3. The Solution: The "AutoMUP" Magic Machine

The researchers didn't just pick one random summary. They built a system called AutoMUP (Automatic Meaning Unit Pyramid) to find the "best" summary automatically.

Here is how AutoMUP works, step-by-step:

Step 1: Breaking it Down (The LEGO Analogy)
Imagine taking all 50 soup descriptions and breaking them down into tiny LEGO bricks. Each brick is a single idea or "meaning unit" (e.g., "The soup needs salt," "The carrots are soft").
Step 2: Grouping by Similarity (The Sorting Hat)
Since one person might say "add salt" and another says "season with salt," the system uses a smart sorting algorithm to realize these are the same LEGO brick. It groups all similar ideas together into piles.
Step 3: The Popularity Contest (The Consensus Weight)
This is the most important part. The system counts how many people mentioned each pile of bricks.
- If 45 out of 50 people mentioned "salt," that pile gets a high score.
- If only 2 people mentioned "a pinch of pepper," that pile gets a low score.
Step 4: Building the Gold Summary
The system builds the "Gold Summary" (the best possible answer) using only the most popular bricks. It creates a summary that represents what the majority of humans agreed was important.

4. The Results: Did it Work?

The researchers tested their "Gold Summary" against summaries written by super-smart AI models (like GPT-5.1 and Flash 2.5).

The Match: The human-made "Gold Summary" matched the AI summaries very closely. This proves that the AutoMUP method is reliable and that the AI is learning the right things.
The "What If" Test: They tried removing the "popularity contest" part of their system. Without it, the summaries became messy and less accurate. This proved that agreement among humans is the secret sauce to making a good summary.

5. Why This Matters

For Turkish Speakers: Turkish is a complex language where words change shape a lot. This study shows that you can create high-quality educational tools for Turkish speakers without needing expensive human experts to manually grade every video.
For the Future: This method is cheap and repeatable. You can use it for other languages (especially other Turkic languages) to build better educational tools.
The "Gold Standard": Now, researchers have a reliable "answer key" to test if new AI summarizers are actually smart or just guessing.

In a Nutshell

Imagine you want to know the "definitive" story of a movie, but you don't want to watch it. Instead of asking one person, you ask a hundred. You then ignore the weird opinions of the few and focus on the parts of the story that everyone agreed were important. AutoMUP does exactly that for educational videos, turning a chaotic crowd of opinions into a single, clear, and trustworthy summary.

1. Problem Statement

The paper addresses several critical gaps in the field of educational video summarization, specifically for Turkish and Turkic languages:

Lack of Gold Standards: Existing methods for generating "gold-standard" summaries (the reference truth for evaluation) rely heavily on manual human annotation (e.g., Pyramid, LitePyramid), which is costly, time-consuming, and prone to annotator bias.
Limitations of Automated Methods: While Large Language Models (LLMs) can generate summaries, they suffer from hallucinations, lack transparency, and depend on specific model versions or prompts, making them unreliable as a reproducible gold standard.
Language Specificity: Turkish and other Turkic languages possess complex morphological structures and high expressive diversity, making it difficult to model duplicate content. There is a scarcity of datasets and evaluation frameworks tailored to these languages.
Multimodal Complexity: Educational videos contain audio, visual, and spoken language elements. Text-only summaries of transcripts often fail to capture the full context, yet current Turkish-specific resources are limited.

2. Methodology

The authors propose a two-pronged approach: the creation of a new dataset (TR-EduVSum) and a novel automated framework for gold summary generation (AutoMUP).

A. The TR-EduVSum Dataset

Content: 82 Turkish educational lecture videos on "Data Structures and Algorithms" sourced from YouTube.
Human Annotations: 138 computer science students (aged 18–22) independently summarized the videos.
Scale: The dataset contains 3,281 independent human summaries, with 36 to 53 summaries per video.
Preprocessing: Summaries shorter than three sentences were filtered out. Video transcripts were extracted and cleaned using punctuation models.

B. The AutoMUP Framework (Automatic Meaning Unit Pyramid)

AutoMUP is designed to generate a reproducible, unsupervised gold summary by statistically modeling human consensus. It operates in three main stages:

Extraction and Embedding:
- Human summaries are automatically segmented into semantic units (sentences) based on punctuation.
- These units are converted into dense vector embeddings using paraphrase-multilingual-MiniLM-L12-v2 (a multilingual Sentence-Transformer optimized for Turkish).
Clustering-Based Consolidation:
- To handle the high variability in how different humans express the same idea, semantic units are grouped using hierarchical clustering based on cosine distance.
- An automated threshold selection procedure determines the optimal clustering distance.
- Each cluster represents a "consensus unit of meaning." The cluster center is calculated, and the unit closest to the center is selected as the representative unit ( $r_k$ ).
Consensus-Weighted Summary Construction:
- Clusters are ranked by their Support Ratio (the number of distinct human summaries contributing to the cluster divided by the total number of summaries).
- Three disjoint summaries are generated based on this ranking:
  - AutoMUP-1 (Gold Standard): Composed of the top $M$ (fixed at 5) clusters with the highest consensus.
  - AutoMUP-2 & AutoMUP-3: Composed of the next tiers of clusters, used for ablation analysis to study the effect of consensus density.

3. Key Contributions

TR-EduVSum Dataset: The first large-scale dataset for Turkish educational video summarization featuring multiple human summaries per video, enabling robust consensus modeling.
AutoMUP Framework: A fully automated, reproducible method for generating gold summaries without manual annotation. It replaces expensive human aggregation with statistical consensus modeling via embedding clustering.
Semantic Unit-Based Analysis: The study is among the first to apply semantic unit-based methods (inspired by Pyramid but automated) specifically to Turkish educational content.
Generalizability: The framework is largely language-agnostic (given suitable multilingual embeddings) and can be adapted to other Turkic languages at low cost.

4. Experimental Results

The study evaluated AutoMUP-1 against summaries generated by strong LLMs (Flash 2.5 and GPT-5.1) and human summaries using multiple metrics (BERTScore-F1, ROUGE-L, BLEURT, SBERT, SimCSE, USE).

Alignment with LLMs: AutoMUP-1 showed the highest semantic overlap with both Flash 2.5 and GPT-5.1 across all metrics. For instance, with Flash 2.5, AutoMUP-1 achieved a BERTScore-F1 of 0.872, compared to 0.860 and 0.849 for the lower-consensus variants (AutoMUP-2 and -3).
Consensus Density Effect: There was a consistent downward trend in performance from AutoMUP-1 to AutoMUP-3, validating that higher consensus correlates with higher summary quality.
Ablation Studies:
- No-Consensus (Removing weighting): Performance dropped significantly (e.g., SimCSE dropped from 0.742 to 0.488), proving that consensus weighting is the primary driver of content representativeness.
- No-Clustering (Removing clustering): Performance improved over "No-Consensus" but remained below AutoMUP-1 (e.g., BERTScore-F1 0.538 vs. 0.574). This confirms that clustering is essential for reducing redundancy and selecting expression-level consistent units.

5. Significance and Conclusion

Reproducibility: AutoMUP offers a transparent, reproducible alternative to LLM-based gold standards, eliminating hallucination risks and prompt dependency.
Cost-Effectiveness: It automates the "Pyramid" method, removing the need for expensive expert annotators to manually extract and weight semantic units.
Turkish NLP Advancement: The paper fills a critical resource gap for Turkish NLP, providing a dataset and evaluation framework that accounts for the language's morphological complexity.
Future Impact: The framework allows for the creation of high-quality gold standards for educational video summarization in low-resource languages, facilitating better training and evaluation of multimodal summarization models.

Limitations: The framework relies on majority consensus, which may overlook minority but valid viewpoints. Additionally, the dataset is currently limited to "Data Structures and Algorithms" lectures, though the methodology is generalizable.