BOOM: Beyond Only One Modality KIT's Multimodal Multilingual Lecture Companion

1. Problem Statement

The globalization of education and the rise of online learning have created a critical need for localizing educational content. However, lecture materials are inherently multimodal, consisting of:

• Audio: Spoken lectures.
• Visuals: Presentation slides containing text, diagrams, formulas, and annotations.
• Text: Transcripts and summaries.

Existing localization systems often fail to address this complexity holistically. Most Speech Translation (ST) systems are unimodal (audio-only), ignoring the visual context provided by slides, which leads to ambiguity in technical terms. Furthermore, existing tools rarely localize the slides themselves, particularly text embedded within images (e.g., diagram labels), leaving non-native speakers with a fragmented learning experience where the audio is translated but the visual aids remain in the original language.

2. Methodology: The BOOM System

The authors propose BOOM, an end-to-end multimodal multilingual lecture companion that processes and translates content across three modalities: translated text, localized slides, and synthesized speech. The system architecture consists of three main pipelines:

A. Multimodal Speech Translation (ST) Pipeline

• Model: Utilizes OmniFusion, a multilingual multimodal ST model (based on Qwen Omni 2.5 7B), fine-tuned on noisy lecture data.
• Context Integration: To provide visual grounding, the system extracts screenshots of the current slide displayed during specific audio segments. These images are fed into the ST model alongside the audio to disambiguate spoken content (e.g., distinguishing between "Exit" for a door vs. a highway).
• Streaming: Implements a low-latency policy combining Voice Activity Detection (VAD) and Local-Agreement to handle live streaming audio.
• Downstream Processing: The multimodal transcripts are segmented into chapters and used for:
  • Summarization: Generating compressed representations (50%, 70%, 90%) and length-controlled summaries.
  • Question Answering (QA): Using Retrieval-Augmented Generation (RAG) to answer student queries based on the translated transcript.

B. Slide Translation Pipeline (Image-to-Image)

To localize slides containing text embedded in images, the authors developed a modular, open-source pipeline:

1. Optical Character Recognition (OCR): Uses PaddleOCR v5 to detect text regions and bounding boxes within slide images.
2. Layout Analysis: Employs the Hi-SAM model to group OCR boxes into coherent semantic blocks (lines, paragraphs, bullet points) and preserve structural cues (font size, color, grouping).
3. Multimodal Translation: Translates the extracted text blocks using OmniFusion, leveraging the original slide image as visual context to ensure accurate translation of domain-specific terms.
4. Inpainting: Uses Simple-LaMa to remove the original text from the image background, reconstructing the background seamlessly.
5. Rendering (Drawing): A heuristic-based module re-renders the translated text into the original layout, preserving alignment, font style, and spatial organization without relying on unstable diffusion-based generation.

C. User Interface

A custom PDF viewer allows users to view slides with synchronized captions. It supports out-of-sync browsing and real-time alignment, enabling students to follow the lecture in their native language while viewing the localized slides.

3. Key Contributions

1. Integrated Multimodal ST: Adapted OmniFusion to leverage slide screenshots during live translation, demonstrating that visual context significantly improves translation accuracy for technical content.
2. Open-Source Slide Localization: Introduced a fully functional, modular pipeline for translating text inside slide images and re-rendering them, a task previously neglected by standard ST tools.
3. Downstream Task Evaluation: Demonstrated that slide-aware transcripts yield cascading benefits for downstream NLP tasks (Summarization and QA) across various Large Language Models (LLMs).
4. Comprehensive Benchmarking: Evaluated OCR models, translation quality (BLEU, ChrF, COMET), and latency across different language pairs and segmentation strategies.

4. Experimental Results

The system was evaluated using the VISTRA benchmark (for image translation) and the MCIF dataset (for downstream tasks).

• OCR Performance: PaddleOCR v5 was selected as the optimal balance between accuracy and latency (0.1s per image), outperforming EasyOCR and Qwen-2.5-VL (which had high latency).
• Translation Quality:
  • OmniFusion consistently outperformed unimodal LLMs (Tower 7B, SeedX) in most language pairs (German, Spanish, Russian, Chinese).
  • Block-level segmentation (grouping lines into semantic units) significantly outperformed line-level segmentation, confirming the need for coherent context.
  • Visual Context: The multimodal approach successfully disambiguated words like "Exit" (pedestrian vs. car) based on the slide image, a capability unimodal models lacked.
• Downstream Tasks (Summarization & QA):
  • Summarization: Using Audio + Image input consistently improved BERTScore across English, German, and Italian compared to Audio-only input.
  • QA: Visual context provided slight improvements, though gains were less pronounced than in summarization. The authors note that LLMs only receive the textual context improved by multimodality, not the raw image, which limits QA gains.
• Latency: The most time-consuming steps were Layout Analysis and Translation. OCR and Inpainting were relatively fast.

5. Significance and Future Work

• Significance: BOOM represents a shift from "audio-only" lecture translation to a holistic multimodal localization approach. By preserving the visual integrity of slides and leveraging them for translation, it significantly enhances accessibility for non-native speakers and improves the quality of downstream AI tasks like summarization.
• Limitations:
  • The slide translation evaluation relies on the VISTRA benchmark, which may not fully reflect the complexity of academic lecture slides.
  • Human evaluation of the visual coherence of rendered slides is needed.
  • Current QA evaluation is post-hoc (after the full talk), not simulating a real-time live scenario.
• Future Work: The authors plan to conduct human evaluations to assess the quality of rendered slides and audio, and to refine the system for real-time interactive scenarios.

In conclusion, BOOM provides a robust, open-source framework for making educational content truly accessible by bridging the gap between spoken language, visual aids, and written text in a multilingual context.