When Meaning Travels: A Granular Lens on Hybrid-MoE's Role in Idiomatic Understanding for Language Models

Technical Summary: When Meaning Travels: A Granular Lens on Hybrid-MoE's Role in Idiomatic Understanding for Language Models

Problem Statement
Idioms serve as a critical conduit for cultural values, historical context, and creative expression, yet they present a persistent challenge for computational modeling. Their meanings are non-compositional, deeply rooted in cultural grounding, and often resist literal interpretation. While Large Language Models (LLMs) and Vision-Language Models (VLMs) have advanced, accurately decoding culture-specific lexical patterns—particularly in low-resource Southeast Asian languages like Hindi, Bengali, and Thai—remains difficult. Existing benchmarks (e.g., FLUTE, V-FLUTE) often lack the multimodal depth and cultural nuance required for these languages, and standard dense architectures are computationally expensive. Furthermore, relying solely on top-k expert selection in Mixture-of-Experts (MoE) frameworks may constrain the expressiveness needed for complex pragmatic tasks like idiom explanation.

Methodology
The authors propose a two-pronged approach: the construction of a new dataset and the development of a novel architectural framework.

1. Varnika Dataset: The authors reconstruct and augment the existing Mediom corpus to create Varnika, a multimodal idiom corpus containing 3,533 instances across Hindi, Bengali, and Thai. Unlike previous resources, Varnika enriches each idiom with seven pragmatic "idiomatic tones": Humor, Ridicule, Affection, Aspiration, Fear, Sorrow, and Deception. The dataset includes paired text and visual representations, curated from literary and archival sources, and validated through a two-stage process involving peer review and expert cultural validation (achieving a Cohen's kappa of 0.65).

2. HybridMoE Framework: To address the complexity of idiomatic understanding, the paper introduces HybridMoE, a multimodal hypernetwork-based architecture.

   • Architecture: The model encodes text (via BERT) and images (via ViT) into embeddings. These are enhanced by Idiomatic Property Signals (IPS), which selectively condition cross-modal embeddings to capture idiom-specific cues.
   • Hybrid Expert Routing: The framework employs three parallel Hybrid-MoE modules, each containing four expert networks. Unlike standard MoE, which routes tokens to a sparse subset of experts, HybridMoE utilizes a controlled hybridization strategy. It integrates outputs from both selected and unselected experts via a trainable gating function and a mean-pull strategy, mitigating expert sparsity while enabling cross-expert knowledge sharing.
   • Evaluation Metrics: The authors propose two novel metrics:
     • Idiomatic Validation (IV) Score: A three-stage pipeline measuring (i) literal translation fidelity (BLEU), (ii) visual-semantic alignment (CLIP cosine similarity), and (iii) idiomatic meaning retention (BERTScore). The final score is the geometric mean of these components.
     • IDIO-TONE Score: A label-based metric using a distilled DeepSeek-R1 model to evaluate the alignment between predicted and ground-truth idiomatic tone labels (F1-score).

Key Contributions

• Varnika Dataset: A multimodal resource for low-resource languages (Hindi, Bengali, Thai) featuring fine-grained tonal annotations and visual grounding, addressing a gap in existing cross-lingual idiom datasets.
• HybridMoE Architecture: A novel framework that combines expert-specific modules with a globally shared hypernetwork and IPS to enhance reasoning over idiomatic constructs.
• Evaluation Framework: The introduction of the IV Score and IDIO-TONE Score to systematically quantify literal fidelity, visual alignment, and pragmatic tone consistency.
• Empirical Validation: Comprehensive benchmarking across multiple VLMs (e.g., Qwen2.5-VL-7B, Gemma3, SmolVLM) demonstrating the efficacy of the proposed framework.

Results
Empirical evaluations indicate that the HybridMoE framework yields significant performance gains over standard fine-tuned VLMs.

• Performance Gains: When applied to Qwen2.5-VL-7B-Instruct, HybridMoE improved ROUGE-L from 0.36 to 0.41, BLEU-3 from 16.51 to 21.48, and BERTScore from 0.93 to 0.95.
• IV Score: The Qwen2.5-VL-7B + HybridMoE configuration achieved the highest IV Score of 0.82, outperforming standard fine-tuning baselines.
• Generalizability: The framework demonstrated architecture-agnostic improvements, with gains observed across diverse models including SmolVLM and Gemma3. Language-specific models (e.g., Ganga for Hindi, Typhoon for Thai) also showed the ability to infer idiomatic meanings when trained on Varnika.
• Limitations in Tone: While semantic and literal metrics improved, IDIO-TONE scores remained moderate (ranging from ~0.30 to 0.55), indicating that capturing nuanced pragmatic tones remains a significant challenge even for advanced models.

Significance and Claims
The paper claims that HybridMoE, integrated with IPS, effectively enhances the representation of figurative language and culturally embedded meanings in multilingual multimodal settings. The authors posit that their work bridges a gap in low-resource language processing by providing a dataset that explicitly links visual grounding with cultural tonality. The results suggest that dynamic expert routing and cross-modal grounding can improve the disentanglement of semantic, cultural, and contextual representations, offering a pathway toward more inclusive and culturally aware AI. However, the authors modestly acknowledge that while HybridMoE improves abstraction, it does not fully resolve the gap between cultural reasoning and perceptual grounding for highly abstract idioms lacking clear visual metaphors.