FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings

1. Problem Statement

Sentence embedding models (encoders) like SONAR, LaBSE, and Gemini compress complex semantic information from text and speech into single, high-dimensional vectors. While these vectors are effective for downstream tasks (retrieval, classification), they function as "black boxes."

• The Gap: There is a lack of comprehensive tools to interpret what specific lexical content or concepts are encoded within these vectors, particularly across different modalities (speech vs. text) and languages.
• Limitations of Current Methods: Existing interpretability research often focuses on English-only text or relies on heavy downstream benchmarking (e.g., MTEB) which ranks models but does not explain why a model performs a certain way or what specific information is lost during compression.
• Goal: The authors aim to develop a diagnostic tool to recover lexical content from embeddings, thereby uncovering biases (modality and language) and quantifying the "linearity" of semantic concepts within the embedding space.

2. Methodology: Factorized Linear Projection (FLiP)

The core contribution is FLiP, a model designed to treat interpretation as a keyword extraction task via linear projection.

A. Core Formulation

Instead of treating the embedding as a black box, FLiP attempts to reconstruct the original "bag-of-words" (vocabulary distribution) from the sentence embedding using a linear projection.

• Input: A sentence embedding vector $t_n$ (from text) or $s_n$ (from speech).
• Target: A bag-of-words vector $x_n$ representing the vocabulary counts of the source text.
• Mechanism: The model learns a projection matrix $W$ and bias $b$ to map the embedding to the vocabulary space. The probability distribution over words is computed as:
  $$\theta_n = \text{softmax}(b + W t_n)$$
• Optimization: The model maximizes the regularized log-likelihood of the training data.

B. Factorization (The "FLiP" Innovation)

The authors introduce a factorized version of the projection matrix $W = AB$, where:

• $A \in \mathbb{R}^{|V| \times r}$: Word embedding matrix.
• $B \in \mathbb{R}^{r \times d}$: Modality-to-latent projection.
• $r$: A low-rank dimension ($r \leq d$).
• Benefit: This factorization acts as an implicit regularizer, improving generalization and training efficiency. It allows the model to achieve high performance with fewer parameters (e.g., a 512-dimension rank performs nearly as well as full-rank 1024).

C. Cross-Modal and Cross-Lingual Training

To handle multimodal (speech/text) and multilingual data, the training objective combines signals from different sources:
$$L = \sum_{n=1}^{N} [\alpha x_n^T \log(\theta_n) + (1-\alpha) x_n^T \log(\phi_n)] - R(W)$$

• $\alpha$ balances the contribution of text and speech (or primary and secondary languages).
• This allows the model to learn a unified representation that is robust to modality shifts (e.g., training on speech to decode text embeddings).

D. Inference

At inference, given an unseen embedding, the model computes logits $z = b + A(Bu)$ and selects the top-$k$ indices as keywords. No search or optimization is required at runtime.

3. Key Contributions

1. High-Fidelity Recovery: Demonstrated that FLiP can recover 75–80% of the lexical content from well-encoded sentence embeddings, proving that semantic concepts are largely linearly represented in these spaces.
2. Diagnostic Tool: Established FLiP as a method to systematically analyze modality alignment (speech vs. text) and language alignment (cross-lingual biases) without relying on downstream task benchmarks.
3. Superiority over Baselines: Showed that FLiP significantly outperforms existing baselines like SpLiCE (which decomposes embeddings into sparse concepts) in terms of keyword extraction accuracy and parameter efficiency.
4. Bias Discovery: Uncovered that while intra-language modality alignment is strong, cross-lingual alignment is heavily English-biased, with performance degrading significantly for linguistically distant languages.

4. Experimental Results & Analysis

A. Factorization and Rank

• Factorization is crucial: Factorized models (FLiP) significantly outperform non-factorized linear probing (LiP).
• Rank Efficiency: A rank of 512 achieved 76.77% accuracy on English text (SONAR), nearly matching the full-rank (1024) model (77.29%) but with half the parameters. Lower ranks (128) saw a drop in performance.

B. Modality Alignment (SONAR)

• Strong Alignment: When trained on one modality (e.g., speech) and tested on the other (text), performance remained high.
• Transferability: Training on speech embeddings slightly improved transferability to text domains compared to the reverse, suggesting speech embeddings in SONAR are robust.
• Consistency: High Jaccard indices (>87%) between models trained on different modalities indicate they extract the same semantic concepts.

C. Language Alignment & Bias

• English Dominance: English (EN) is the "preferred" language for linear representation.
  • Training on EN yields high accuracy on linguistically similar languages (German, French).
  • Training on non-EN languages results in a significant performance gap when tested on EN.
• Distant Languages: For linguistically distant pairs (e.g., English-Tamil, English-Telegu), the performance gap widens drastically (e.g., dropping from ~70% to ~48% when vocabularies are mismatched), indicating poor cross-lingual linear alignment in the embedding space.
• Vocabulary Effect: Using an English vocabulary yields the best results regardless of the input language, further highlighting the English-centric nature of these embeddings.

D. Comparison Across Encoders

• SONAR vs. LaBSE vs. Gemini:
  • SONAR (multimodal) showed the best keyword extraction performance across all languages.
  • LaBSE (text-only) and Gemini (API-based) performed lower, with Gemini showing significant drops when the vocabulary language did not match the embedding language.

E. Named Entity Recall

• Bias Correction: Removing the bias vector $b$ (which acts as a log-prior for frequent stop words) actually improved Named Entity recall. The bias tends to up-weight common stop words (e.g., "was", "in"), displacing specific entities in the top-$k$ list.

F. Comparison with SpLiCE

• FLiP achieved ~61% span-aware accuracy on text embeddings, nearly double the performance of SpLiCE (~29%), proving it is a more effective tool for interpreting dense embeddings.

5. Significance and Conclusion

• Interpretability: FLiP provides a "scientific pursuit" into what is crammed into a single vector, confirming that linear projections are sufficient to decode most lexical content.
• Practical Utility: It offers practitioners a lightweight, intrinsic diagnostic tool to evaluate model biases (e.g., "Does my model understand Tamil speech as well as English text?") without the cost of massive downstream benchmarking.
• Implications for Model Design: The findings suggest that while current models are excellent at aligning modalities within a language, they suffer from significant English bias in cross-lingual settings. Future models need to improve linear semantic alignment for low-resource and linguistically distant languages.

The code for FLiP is publicly available, enabling the community to further probe and improve multimodal multilingual embeddings.