💬 NLP

Building a Multimodal Dataset of Academic Paper for Keyword Extraction

This study addresses the scarcity of multimodal resources for keyword extraction by constructing a new dataset of 1,000 academic papers containing text, images, and audio, demonstrating that fusing information from these diverse modalities significantly improves extraction performance compared to using text alone.

Original authors: Jingyu Zhang, Xinyi Yan, Yi Xiang, Yingyi Zhang, Chengzhi Zhang

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingyu Zhang, Xinyi Yan, Yi Xiang, Yingyi Zhang, Chengzhi Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find the most important ingredients in a giant, complex recipe book. Usually, when people try to pick out the key ingredients (which researchers call "keywords"), they only read the written text of the recipe. They ignore the pictures of the finished dish or the audio recording of the chef explaining the steps.

This paper argues that by ignoring the pictures and the audio, we are missing out on a lot of flavor. The authors built a new "kitchen" (a dataset) to test if looking at the whole meal—text, images, and audio together—helps us identify the key ingredients better.

Here is a simple breakdown of what they did and what they found:

1. The Problem: A One-Sided Conversation

For a long time, computers have been very good at reading text to find keywords. But in the real world, information isn't just text. It's a mix of:

Text: The actual words on the page.
Images: Slides, charts, and photos.
Audio: The sound of a speaker talking.

The authors say that by only listening to the "text" part of the conversation, we miss the clues hidden in the pictures and the voice. Also, there wasn't really a "recipe book" available that had all three of these things lined up together for researchers to study.

2. The Solution: Building a New "Recipe Book"

To fix this, the team created a brand-new dataset called a Multimodal Dataset.

What's inside? They gathered 1,000 samples from academic conferences.
The ingredients: For every single sample, they collected:
1. The written paper (the text).
2. The presentation slides (the images).
3. The recording of the speaker (the audio).
4. The list of keywords the authors originally chose (the "answer key").

Think of this like creating a study guide where every page has the essay, the diagrams, and a recording of the teacher explaining it, all perfectly synced up.

3. The Experiment: The Taste Test

Once they had their dataset, they ran a "taste test" using different computer programs (models) to see which method could find the keywords best. They tested three scenarios:

The Text-Only Diet: Feeding the computer only the written paper.
The Audio & Image Diet: Feeding the computer only the text transcribed from the audio or the text recognized from the images (using OCR, which is like a robot reading a sign).
The Buffet: Feeding the computer a mix of all three text sources combined.

4. The Results: What Worked Best?

Here is what the "taste test" revealed:

The Written Paper is the Star: The text from the actual academic papers was the most reliable source. It had the richest information, so the computers did the best job finding keywords here.
Audio is a Good Side Dish: The text transcribed from the speaker's voice was helpful, though not quite as good as the written paper.
Images are the Tricky Part: The text pulled from the slides (images) performed the worst. The authors explain this is like trying to read a menu written on a blurry, crumpled napkin; the computer often got confused by the background noise or bad lighting, making the text hard to read.
The Power of the Buffet (Fusion): The biggest discovery was that combining all three sources worked better than using just one. Even though the image text was messy, when the computer looked at the paper, the audio, and the image text all at once, it could fill in the gaps. If a keyword was missing from the paper but mentioned in the audio, the "Buffet" method caught it.

5. The Takeaway

The main lesson from this paper is that while the written text is the most important ingredient, ignoring the other parts of the presentation (the voice and the slides) leaves information on the table.

By building this new dataset and proving that mixing these different types of information together improves the computer's ability to find keywords, the authors have given researchers a new tool. It's like upgrading from reading a recipe in the dark to reading it with a flashlight, a picture of the dish, and a recording of the chef all at once.

In short: They built a new library of mixed-media academic papers and proved that when computers read the text, listen to the audio, and look at the slides all together, they get a much clearer picture of what the paper is actually about.

Technical Summary: Building a Multimodal Dataset of Academic Paper for Keyword Extraction

Problem Statement
Current keyword extraction tasks predominantly rely on textual data, often neglecting visual details and audio features inherent in multimodal academic content. This reliance leads to deficiencies in information richness and overlooks potential correlations between modalities, constraining a model's ability to learn accurate data representations. Furthermore, the research field suffers from a scarcity of multimodal datasets specifically designed for keyword extraction, which hinders the development and validation of multimodal extraction methods. While academic information is increasingly disseminated through text, images (slides), and audio (lectures), existing methods fail to leverage these diverse sources effectively.

Methodology
To address these gaps, the study constructs a new multimodal dataset and evaluates keyword extraction performance across different modalities and fusion strategies.

Dataset Construction: The authors curated a dataset of 1,000 academic paper samples sourced from VideoLectures.net and the SPIE Digital Library. Each sample includes:
- Paper Text: Full text of the academic paper.
- Slide Images: Presentation slides extracted from conference videos.
- Audio: Speaker recordings from the presentations.
- Keywords: Author-provided keywords serving as the ground truth.
- Preprocessing: The pipeline involved web scraping, video segmentation (using HLS/DASH protocols), audio extraction (OpenCV), frame deduplication via structural similarity algorithms, and Optical Character Recognition (OCR) for slide text. Audio transcripts were obtained via Automatic Speech Recognition (ASR). All modalities were aligned based on timestamps. Data cleaning involved removing garbled characters and formatting inconsistencies.
Experimental Setup: The study employed both unsupervised and supervised keyword extraction models:
- Unsupervised: TF-IDF, TextRank, and SVM (with under-sampling to balance classes).
- Supervised: CRF, BiLSTM-CRF, and BERT-BiLSTM-CRF.
- Input Variations: Experiments were conducted using text derived solely from papers ( $P_t$ ), audio ( $A_t$ ), and images ( $I_t$ ), as well as a concatenated multimodal text ( $M_t$ ) combining all three sources.
- Evaluation: Performance was measured using Precision, Recall, and the F1 score (F1@3, F1@5, F1@10) based on exact matching against author-provided keywords.

Key Results

Modality Performance: Paper text consistently yielded the best keyword extraction performance across most models, particularly in supervised approaches like CRF and SVM. Audio text performed moderately well, while image text (derived from OCR) performed the worst. The authors attribute the poor performance of image text to noise and background interference affecting OCR accuracy, and audio text limitations to recording quality and background noise.
Fusion Effectiveness: The concatenation of paper, audio, and image text ( $M_t$ ) effectively enhanced keyword extraction performance in several models, particularly deep learning architectures like BiLSTM-CRF and BERT-BiLSTM-CRF. This suggests that information from different modalities can complement one another; for instance, keywords missed in the paper text were sometimes recoverable from audio or image text.
Model Comparison: Deep learning models (BiLSTM-CRF, BERT-BiLSTM-CRF) generally outperformed traditional statistical methods (TF-IDF, TextRank) when utilizing concatenated multimodal data. However, traditional models like CRF and SVM remained robust, with CRF showing strong performance on paper text alone.

Key Contributions

Dataset Creation: The study fills a critical gap by providing the first multimodal dataset specifically tailored for academic keyword extraction, comprising 1,000 high-quality samples with aligned text, image, and audio modalities.
Empirical Validation: The research demonstrates that concatenating text from paper, image, and audio modalities can improve keyword extraction accuracy compared to using single modalities.
Methodological Insight: The work highlights the distinct characteristics of different modalities, noting that while paper text is the primary source of keywords, multimodal fusion offers a more comprehensive semantic capture, potentially aiding models in overcoming the limitations of single-source data.

Significance and Future Directions
The paper claims that its primary significance lies in providing new data resources for multimodal academic research and advancing keyword extraction toward the utilization of broader modal data, thereby enhancing algorithm accuracy and robustness. The authors maintain a modest stance, acknowledging that their current experiments utilized only the textual content extracted from the modalities, ignoring unique features such as visual layout (font size, bolding) in slides or acoustic features (speech energy, duration) in audio. They suggest that future work should explore these specific modality features and leverage the self-learning and generalization capabilities of large language models to further improve multimodal keyword extraction.

1. The Problem: A One-Sided Conversation

2. The Solution: Building a New "Recipe Book"

3. The Experiment: The Taste Test

4. The Results: What Worked Best?

5. The Takeaway

More like this