OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusion

1. Problem Statement

The paper addresses the limitations of current translation systems in handling multimodal (text, audio, image) and multilingual tasks, particularly in Simultaneous Speech Translation (SimulST) scenarios.

• Limitations of Text-Only LLMs: While open-source Large Language Models (LLMs) excel at multilingual text translation, their unimodal design prevents them from utilizing visual context (e.g., presentation slides, images) to disambiguate speech or text. In speech translation, they typically rely on cascaded pipelines (Automatic Speech Recognition $\to$ Machine Translation), which introduces significant latency and prevents the translation model from leveraging audio-visual cues simultaneously.
• Limitations of Multimodal Foundation Models (MMFMs): Pretrained MMFMs possess strong cross-modal perception and reasoning but generally lack the deep multilingual translation capabilities and language coverage of specialized translation LLMs, resulting in lower translation quality.
• The Gap: There is a need for a unified, end-to-end (E2E) architecture that combines the perceptual strengths of MMFMs with the translation proficiency of specialized LLMs to achieve low-latency, high-quality, context-aware translation.

2. Methodology: OmniFusion Architecture

The authors propose OmniFusion, a unified architecture that fuses a Multimodal Foundation Model (MMFM) with a translation-specialized LLM via a lightweight gated fusion module.

Core Components

• MMFM Backbone: Uses Qwen2.5-Omni-7B to process audio and image inputs, extracting perceptual representations.
• Translation LLM Backbone: Uses Seed-X-PPO-7B for multilingual understanding and generation.
• Gated Fusion Module: Instead of concatenating all MMFM layers (which increases sequence length and computational cost), the model extracts hidden states from three specific layers of the MMFM:
  1. First Layer: Captures low-level perceptual features.
  2. Middle Layer: Captures intermediate abstract reasoning.
  3. Last Layer: Captures high-level semantic understanding.
  • A gating mechanism ($W_{gate}$) dynamically weights these three representations based on the input token, sums them, and projects them via an MLP to align with the translation LLM's embedding dimension. This allows the model to selectively integrate multimodal context without bloating the sequence.

Training Strategy

• Multi-Task Learning: The model is trained on three primary tasks:
  1. Speech Translation (ST): Audio $\to$ Text.
  2. Speech-Image Translation (SIT): Audio + Image $\to$ Text.
  3. Text-Image Translation (TIT): Text + Image $\to$ Text (e.g., caption translation).
• Self-Cascading: To improve alignment between modalities and the LLM's semantic space, the training employs a stochastic self-cascading approach. The model is prompted to first generate intermediate outputs (ASR transcription for audio, OCR text for images) before generating the final translation. This is done using specific prompts (e.g., "Transcribe the audio using the image for context...").
• Parameter Efficiency: The MMFM weights are frozen, and only the translation LLM is fine-tuned using LoRA (Low-Rank Adaptation). This reduces computational costs and prevents catastrophic forgetting of the MMFM's multimodal capabilities.

Inference Modes

• Direct Translation: End-to-end generation from multimodal input to target text.
• Self-Cascade Inference: The model explicitly generates the intermediate ASR/OCR step before the final translation, improving quality at the cost of slightly higher latency.
• SimulST Policy: For simultaneous translation, the authors adopt the Local Agreement policy, which commits the longest common prefix between consecutive hypotheses to minimize latency.

3. Key Contributions

1. Unified Architecture: Introduction of OmniFusion, the first E2E model to jointly support speech, text, and image translation using a fusion of a specialized translation LLM and a multimodal foundation model.
2. Gated Fusion Mechanism: A novel, efficient method to integrate specific MMFM layers into a translation LLM, enabling context-aware translations without the computational overhead of processing all layers or concatenating full sequences.
3. Performance Gains: Demonstration that OmniFusion reduces SimulST latency by approximately 1 second compared to cascaded pipelines while maintaining competitive translation quality. It also significantly reduces critical and major translation errors in offline settings.

4. Experimental Results

The model was evaluated on datasets including MCIF (scientific talks with slides), FLEURS (multilingual speech), and CoMMuTE (image-caption translation).

• Simultaneous Speech Translation (SimulST):
  • OmniFusion achieved a ~1 second reduction in Average Lagging (latency) compared to cascaded baselines (ASR $\to$ MT) across various chunk sizes.
  • When visual context (images) was provided, OmniFusion achieved both the lowest latency and highest quality, proving that the E2E model exploits visual cues more effectively than cascaded systems.
• Offline Speech Translation:
  • OmniFusion with Gated Fusion + Self-Cascade achieved an XCOMET-XL score of 86.57, nearly matching the best cascaded LoRA-fine-tuned baseline (86.59).
  • Crucially, the E2E model produced fewer critical and major errors (e.g., 55.3 critical errors vs. 75.0 for the best cascaded system), indicating higher reliability.
• Text-Image Translation (Captioning):
  • On the CoMMuTE benchmark, OmniFusion (14B parameters) achieved State-of-the-Art (SOTA) results, outperforming or matching vision-only models like ZeroMMT and TowerVision.
  • It successfully disambiguated words based on visual context (e.g., distinguishing "EXIT" as a vehicle exit vs. a person exit based on the image).
• Ablation Studies:
  • Layer Selection: Using only the First, Middle, and Last layers was empirically validated as optimal. Using all layers or attention-based fusion introduced noise and degraded performance.
  • Gate Behavior: Analysis showed the gating mechanism assigns the highest weights to the first and middle layers, suggesting that early and intermediate MMFM layers contain the most transferable multimodal information.

5. Significance and Impact

• Efficiency: By freezing the MMFM and using a lightweight gating mechanism, OmniFusion offers a computationally efficient path to multimodal translation, avoiding the need for massive-scale pretraining from scratch.
• Latency Reduction: The reduction in SimulST latency is critical for real-world applications like live conference translation, where delays in cascaded systems are unacceptable.
• Error Reduction: The ability to reduce critical errors suggests that E2E multimodal models are more robust in handling ambiguity than modular pipelines.
• Generalizability: The approach demonstrates that specialized translation LLMs and multimodal foundation models can be effectively combined, offering a blueprint for future multilingual multimodal systems.

Limitations: The current training data is primarily English-centric, which may limit performance for other source languages. Additionally, the model relies on fixed task-specific prompts and does not yet support general instruction-following across arbitrary multimodal tasks. Future work aims to expand multilingual coverage and explore instruction-driven multimodal behavior.