Using Multimodal and Language-Agnostic Sentence Embeddings for Abstractive Summarization

This paper introduces SBARThez, a novel framework that leverages multimodal and multilingual sentence embeddings alongside a Named Entity Injection mechanism to enhance the factual consistency and cross-lingual capabilities of abstractive summarization for both text and speech inputs.

The Gist

Imagine you have a massive library of books, podcasts, and news reports in dozens of different languages. You want to know the main points of a 50-page article or a 20-minute conversation, but you don't have time to read or listen to the whole thing. You need a summary.

For a long time, computers tried to do this by acting like photocopiers. They would scan the original text, pick out the "best" sentences, and glue them together. This is called extractive summarization. It's safe, but it's not very creative.

Then, computers got smarter and started acting like writers. They would read the whole thing and write a brand-new summary from scratch, using their own words. This is abstractive summarization. It's much better, but it has a dangerous flaw: hallucinations.

Think of a hallucination like a student who didn't study for a test but tries to bluff their way through. They might say, "The President visited the moon last Tuesday," even though that never happened. In the world of AI, this means the computer invents facts, names, or places that weren't in the original story.

The New Solution: SBARThez

The authors of this paper built a new system called SBARThez (a playful name combining "Sentence," "BART," and "Thez"). Here is how it works, broken down into simple concepts:

1. The "Gist" vs. The "Letters"

Most AI models read text like a human reads a book: letter by letter, word by word. They focus on the tiny details (tokens).

• The Old Way: Imagine trying to understand a movie by looking at every single pixel on the screen. It's detailed, but you might miss the big picture.
• The SBARThez Way: This model looks at the sentences as whole blocks of meaning. It uses special "sentence embeddings." Think of these as summary cards. Instead of reading the whole sentence, the model looks at a single card that says, "This sentence is about a politician visiting a school."
• Why it's cool: Because it thinks in "gists" rather than "letters," it can understand the meaning of a sentence in French, Spanish, or even a spoken audio clip, and turn it into a French summary without needing to translate word-for-word first. It's like having a universal translator that understands the idea, not just the dictionary definition.

2. The "Fact-Check" Safety Net (Named Entity Injection)

The biggest problem with these "gist" models is that they are so good at understanding the vibe that they sometimes forget the specific names. They might summarize a story about "Elon Musk" and just say "the famous tech CEO," or worse, invent a new CEO entirely.

To fix this, the authors added a Named Entity Injection mechanism.

• The Analogy: Imagine you are writing a story about a football game. You know the general play-by-play (the gist), but you want to make sure you get the player names right. So, before you start writing, you pull out a list of the players on the field and tape it to your desk.
• How it works: The system scans the original text, pulls out all the important names (People, Organizations, Locations), and feeds them directly to the "writer" part of the AI. This acts as a cheat sheet, forcing the AI to use the real names from the source text, drastically reducing the chance of making things up.

3. Speaking and Listening

This system is also multimodal, meaning it can handle both text and speech.

• The Analogy: Most summarizers are like librarians who only read books. SBARThez is like a librarian who can also listen to a podcast, a phone call, or a lecture, understand the main points, and write a summary.
• It doesn't matter if the input is a typed article or a recording of a conversation; the system converts the audio into those same "summary cards" (embeddings) and processes them just like text.

Why Does This Matter?

1. It's Great for Small Languages: Most AI is trained on English. If you try to summarize a story in a rare language (like Igbo or Kirundi), standard AI often fails or needs to translate it first (which introduces errors). SBARThez works directly on the "meaning" of these languages, making it much better for low-resource languages.
2. It's More Concise: Because it thinks in sentences rather than words, it tends to write shorter, punchier summaries that get straight to the point, rather than just copying chunks of the original text.
3. It's Honest: By using the "Name List" trick, it stops the AI from making up fake facts, which is crucial for news and medical summaries.

The Bottom Line

The authors created a summarizer that doesn't just copy-paste or blindly guess. It reads the "soul" of the content (using sentence embeddings), checks its facts against a list of real names (Named Entity Injection), and can handle both written text and spoken audio. It's like upgrading from a photocopier to a smart, multilingual editor who knows exactly who was in the room and what actually happened.

Technical Summary

Here is a detailed technical summary of the paper "Using Multimodal and Language-Agnostic Sentence Embeddings for Abstractive Summarization."

1. Problem Statement

Abstractive summarization aims to generate concise, coherent summaries by paraphrasing and synthesizing source content rather than simply extracting sentences. However, current state-of-the-art models (typically based on token-level Transformers like BART or T5) face two critical challenges:

• Hallucinations: Models often introduce non-existent information, particularly regarding Named Entities (NEs), leading to factual inconsistencies.
• Modality and Language Limitations: Traditional token-based approaches struggle with cross-lingual tasks (summarizing text in one language into another) and cross-modal tasks (summarizing speech directly into text) due to the scarcity of parallel datasets and error propagation in pipeline systems (e.g., ASR errors affecting summarization).

The paper proposes moving away from discrete token-level processing toward sentence-level semantic representations to improve abstraction, cross-lingual capability, and factual consistency.

2. Methodology: The SBARThez Framework

The authors introduce SBARThez (Semantic-BARThez), a novel architecture that replaces the standard token-based encoder with a sentence-embedding-based encoder while retaining a token-based decoder.

A. Core Architecture

• Sentence Embedding Input: Instead of tokenizing the input document, the model divides the text into sentences. Each sentence is encoded into a dense vector using pre-trained, language-agnostic sentence embedding models (specifically LaBSE, SONAR, and BGE-M3).
• Modified Encoder-Decoder:
  • Encoder: The token embedding layer of a pre-trained French model (BARThez) is removed. A linear projection layer (with GeLU activation) is added to map the varying dimensions of sentence embeddings (e.g., 768 or 1024) to the model's expected input size (768).
  • Decoder: The original token-based decoder of BARThez is preserved to generate the summary token-by-token.
• Two-Stage Training:
  1. Adaptation Stage: The model is trained on the large-scale MLSUM dataset to learn how to process sentence embeddings effectively.
  2. Fine-tuning Stage: The model is fine-tuned on specific task datasets (e.g., OrangeSum for monolingual, WikiLingua/CrossSum for cross-lingual, DECODA for speech).

B. Named Entity Injection (NEI) Mechanism

To address the issue of hallucinated entities, the authors introduce a Named Entity Injection module:

• Extraction: Named entities are extracted from the source document using a French NER model (camembert-ner).
• Injection: These entities (along with their types: PER, ORG, LOC, MISC) are tokenized and appended to the decoder's input as start tokens.
• Goal: This forces the decoder to explicitly consider factual entities from the source during generation, reducing the risk of inventing non-existent names.

C. Multimodal Support

The framework supports Speech-to-Text summarization by replacing text sentence embeddings with speech utterance embeddings derived from models like SAMU-XLSR, SONAR, and SENSE. These models align speech representations with the same semantic space as text embeddings.

3. Key Contributions

1. Novel Architecture (SBARThez): A modified BART-based model that operates on sentence embeddings rather than tokens, enabling efficient processing of multilingual text and speech without tokenization in the encoder.
2. Named Entity Injection (NEI): A mechanism to explicitly inject source entities into the decoder, significantly mitigating hallucination risks.
3. Cross-Modal and Cross-Lingual Versatility: The model demonstrates the ability to handle text-to-text, speech-to-text, and cross-lingual summarization (X $\to$ French) within a single unified framework.
4. Low-Resource Performance: The approach shows superior performance in low-resource language scenarios where traditional translation pipelines or NER systems fail.

4. Experimental Results

A. Monolingual Text Summarization (French $\to$ French)

• Dataset: OrangeSum.
• Performance: SBARThez variants achieved competitive ROUGE-L and BertScore results compared to the token-based BARThez baseline.
• Hallucination Reduction: Without NEI, SBARThez had high Named Entity Hallucination Risk (NEHR) (e.g., 58.52% for BGE variant). With NEI, NEHR dropped to 34.16%, comparable to the ground truth.
• Abstractive Quality: SBARThez models exhibited significantly lower Extractive Fragment Coverage (EFC) and Precision-ROUGE-1 (P-R1) compared to token-based models. This indicates SBARThez generates more abstract summaries (paraphrasing) rather than copying source text, while maintaining high semantic similarity.

B. Cross-Lingual Summarization (X $\to$ French)

• Datasets: WikiLingua and CrossSum.
• High-Resource Languages: SBARThez-BGE achieved competitive BertScores against Translate-Then-Summarize (TTS) pipelines and LLMs (LLaMA-8B).
• Low-Resource Languages: The model significantly outperformed TTS and LLMs in languages like Igbo, Kirundi, and Pidgin.
  • Reasoning: TTS failed due to the lack of Machine Translation models for these languages. SBARThez succeeded because it relies on language-agnostic sentence embeddings that capture semantic meaning without needing explicit translation or NER resources for the source language.

C. Speech Summarization

• Dataset: DECODA (French spoken conversations).
• Results: The model trained with both text and speech embeddings achieved the best performance (BertScore ~35.05).
• Robustness: The model maintained consistent performance across different audio segmentation strategies (Voice Activity Detection, Speaker Diarization, and fixed time windows), proving its viability for real-world applications where ground-truth segmentation is unavailable.
• Comparison: While cascaded systems using high-quality ASR (Whisper Large) performed slightly better, SBARThez outperformed systems using low-quality ASR (Whisper Tiny), demonstrating robustness to transcription noise.

5. Significance and Conclusion

The paper presents a paradigm shift in summarization by decoupling the input representation from token-level processing.

• Efficiency: By operating on sentence embeddings, the model reduces computational overhead associated with tokenization and allows for seamless integration of multimodal inputs.
• Factual Consistency: The NEI mechanism offers a practical solution to the "hallucination" problem, a major bottleneck in abstractive summarization.
• Accessibility: The framework's ability to summarize low-resource languages and speech directly without requiring parallel datasets or intermediate translation steps makes it a powerful tool for democratizing access to information.

The authors conclude that SBARThez sets a new standard for efficient, abstractive, and factually consistent summarization across diverse languages and modalities, opening new avenues for end-to-end speech and cross-lingual summarization research.