🤖 AI

RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval?

This reproducibility study validates that the TRIANGLE framework's geometric triplet alignment significantly improves zero-shot multimodal retrieval performance over pairwise baselines, while also revealing critical optimization instabilities in scratch training and domain-dependent generalization trade-offs.

Original authors: Arijit Ghosh, Aritra Bandyopadhyay, Chiranjeev Bindra, Jingfen Qiao

Published 2026-05-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Arijit Ghosh, Aritra Bandyopadhyay, Chiranjeev Bindra, Jingfen Qiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Three-Way Handshake" Problem

Imagine you are trying to teach a robot to understand the world by showing it three things at once: a Video (what it looks like), Audio (what it sounds like), and Text (a description of what's happening).

For a long time, AI researchers used a "pairwise" strategy. Think of this like a two-person handshake.

The robot shakes hands with the Text and the Video.
Then, it shakes hands with the Text and the Audio.
The Problem: The robot never actually forces the Video and the Audio to shake hands with each other. They might both agree with the Text, but they might still be completely confused about one another. It's like two people agreeing with a third person but hating each other.

The TRIANGLE Solution: The "Triangle Area"

The original paper proposed a new method called TRIANGLE. Instead of just checking two handshakes, it looks at all three at once.

Imagine the three modalities (Video, Audio, Text) are three points floating in space.

Old Way (Cosine Similarity): You just check how close Point A is to Point B, and Point A is to Point C.
TRIANGLE Way: You draw a triangle connecting all three points. The goal is to make the area of that triangle as small as possible.

If the area is tiny, it means all three points are huddled together in a tight group. This forces the Video and Audio to align with each other because they are both trying to get close to the Text. The paper claims this "geometric" approach creates a much tighter, more consistent understanding of the world.

What This Reproducibility Study Did

The authors of this new paper (the "RE-TRIANGLE" team) decided to test if the original claims were true. They acted like independent auditors, trying to rebuild the TRIANGLE robot from scratch to see if it really worked.

Here is what they found, broken down into four key stories:

1. The "Zero-Shot" Success Story (It works, but picky)

The Claim: TRIANGLE is better than the old methods when you give it a brand new dataset it has never seen before.
The Verdict: Mostly True.

The Analogy: Imagine a chef who learned to cook using a specific set of ingredients (the training data). When you ask them to cook a new dish with different ingredients (a new dataset), they do a great job.
The Result: On general, diverse datasets (like random web videos), TRIANGLE was a superstar, beating the old methods by a significant margin (up to 8.7% better).
The Catch: The chef is a bit of a "one-trick pony." When the team tested TRIANGLE on a very specific type of video—cooking tutorials (YouCook2)—it crashed and burned. The old method (VAST) actually did better.
Why? Cooking videos are very repetitive (lots of chopping, stirring). The "triangle" method got confused by this sameness, while the old "fusion" method (which mashes the video and audio together first) handled the repetition better.

2. The "Fine-Tuning" Trap (The Forgetful Student)

The Claim: If you teach TRIANGLE specifically on cooking videos, it should get really good at cooking.
The Verdict: True, but with a side effect.

The Analogy: Imagine a student who is good at math and history. You cram them for a week on only cooking. They ace the cooking test. But when you ask them a math question afterward, they've forgotten everything.
The Result: When the team fine-tuned TRIANGLE on cooking videos, it got much better at finding cooking videos. However, it completely forgot how to handle general videos. Its performance on general datasets dropped drastically. It traded its general knowledge for specific expertise.

3. The "Learning from Scratch" Mystery (The Broken Blueprint)

The Claim: TRIANGLE is so powerful it can learn to understand the world from zero, without any prior training.
The Verdict: Failed to Reproduce.

The Analogy: The original paper handed the team a blueprint for a self-driving car that supposedly works without a driver's license. The team tried to build it from raw metal and wire, but the car wouldn't start.
The Result: When they tried to train the model from absolute zero (no pre-training), it failed miserably. It only worked when they started with a "pre-trained" version (a car that already had a license).
The Diagnosis: They found that a specific part of the math (called the "Data-Text Matching" loss) was unstable when the model was brand new. It was like trying to balance a Jenga tower while the table was shaking. The original authors might have used hidden tricks or settings that the team couldn't find.

4. The "Cosine Regularization" Safety Net

The Claim: Adding a little extra math rule (cosine regularization) helps keep things stable.
The Verdict: True, but only one way.

The Analogy: Think of this as a seatbelt.
The Result: When the robot tries to find a Video using a Text query (Text → Video), the seatbelt works wonders. It stops the robot from getting lost. However, when the robot tries to find Text using a Video query (Video → Text), the seatbelt does almost nothing. The video is already so descriptive that it doesn't need the extra help.

The Final Takeaway

The TRIANGLE idea is brilliant. By forcing the Video, Audio, and Text to form a tight "triangle," it creates a much more unified understanding of the world than older methods, especially for general, diverse content.

However, the study revealed three important warnings:

It's sensitive: It works great on diverse data but struggles on very specific, repetitive data (like cooking shows).
It's fragile: If you try to teach it a new specific skill (fine-tuning), it might forget its old general skills.
It's hard to build from scratch: You can't just start from zero; you need a pre-trained head start, and the original instructions for doing so were incomplete.

In short: TRIANGLE is a powerful tool for aligning different senses, but it requires careful handling and isn't a "one-size-fits-all" magic bullet.

Technical Summary: RE-TRIANGLE

Problem Statement
Multimodal information retrieval aims to bridge the semantic gap between co-occurring signals (e.g., text, video, audio). While foundational models like CLIP and ALIGN revolutionized the field using pairwise contrastive learning, they introduce a "geometric blind spot." Traditional pairwise strategies align peripheral modalities (e.g., video and audio) independently to a central anchor (e.g., text), but they lack explicit constraints to enforce mutual consistency between the peripheral modalities themselves. Fusion-based methods that collapse modalities into a single vector prior to alignment further obscure the fine-grained geometric structure of the latent space. Although recent geometric approaches (e.g., Symile, GRAM) attempt to model joint distributions, they often suffer from computational instability or the "vanishing volume" problem in high dimensions.

Methodology
This paper presents a comprehensive reproducibility study of the TRIANGLE framework, which proposes a tri-modal geometric alignment objective. Instead of relying on independent pairwise cosine similarities, TRIANGLE defines similarity based on the area of a triangle formed by three modality embeddings (Text, Video, Audio) lying on a unit hypersphere.

Geometric Objective: The model minimizes the area of the triangle formed by the three embedding vectors. A smaller area implies the vectors are proximal, indicating strong holistic alignment.
Loss Function: The total loss ( $L_{TOT}$ ) combines directional contrastive losses ( $L_{D2T}$ and $L_{T2D}$ ) that minimize the triangle area for positive triplets and maximize it for negative ones, alongside a Data-Text Matching ( $L_{DTM}$ ) loss for binary classification.
Regularization: A cosine regularization term is added to the triangle area metric to ensure that while the triplet geometry is optimized, task-relevant pairwise alignments are not degraded.
Experimental Design: The authors replicated the original study using pre-trained checkpoints on diverse benchmarks (MSR-VTT, DiDeMo, ActivityNet, VATEX, AudioCaps) and extended the evaluation to the out-of-domain YouCook2 dataset. They also attempted to train the model from scratch on MSR-VTT and a synthetic "toy" dataset of geometric shapes to isolate optimization stability issues.

Key Contributions and Findings

Zero-Shot Retrieval Effectiveness (RQ1):
The study confirms that the tri-modal geometric objective outperforms independent pairwise baselines (specifically VAST) in zero-shot settings across diverse, open-domain datasets. TRIANGLE achieved Recall@1 gains of up to +8.7 points on ActivityNet for Text-to-Video retrieval. However, the benefits are domain-dependent; on the homogeneous, instructional YouCook2 dataset, TRIANGLE significantly underperformed VAST, suggesting the geometric alignment is sensitive to domain-specific semantics.
Domain Adaptation and Generalization (RQ2):
While fine-tuning TRIANGLE on YouCook2 improved in-domain performance, it resulted in catastrophic forgetting of previously learned cross-domain geometry. The model suffered severe performance drops (e.g., ~30 points on MSR-VTT) when applied to out-of-domain datasets after fine-tuning, indicating a trade-off between domain-specific adaptation and global alignment stability.
Learning from Scratch (RQ3):
The authors failed to reproduce the "learning-from-scratch" results reported in the original paper. When training from random initialization without pre-trained weights, TRIANGLE performed poorly, particularly when the $L_{DTM}$ loss was included. Analysis using a synthetic toy dataset revealed that the $L_{DTM}$ objective causes instability when jointly optimized with geometric alignment without pre-trained encoders. TRIANGLE trained without $L_{DTM}$ on the toy dataset significantly outperformed variants with $L_{DTM}$ , suggesting the matching loss requires pre-training to converge effectively.
Interpretability and Regularization (RQ4):
- Cosine Regularization: The study found that adding cosine regularization primarily stabilizes and improves Text-to-Video retrieval, where text queries are more abstract. It had minimal impact on Video-to-Text retrieval, where visual embeddings already provide strong semantic signals.
- Geometric Visualization: PCA visualizations of the embedding space confirmed that the triangle area serves as an interpretable metric: positive triplets exhibit smaller areas than negative ones. However, the visualizations also revealed that audio embeddings often remain geometrically distant from text and video, explaining uneven cross-modal performance.

Significance
The paper validates the core theoretical claim that geometric alignment can enhance multimodal retrieval beyond pairwise cosine similarity, establishing a new state-of-the-art for zero-shot tasks in diverse domains. However, it critically highlights that this approach is not universally superior; its efficacy is contingent on dataset diversity and the direction of retrieval. Furthermore, the study exposes critical optimization sensitivities, specifically the instability of the $L_{DTM}$ loss during training from scratch and the fragility of the learned geometry under domain shifts. The work underscores the necessity of rigorous reproducibility studies to distinguish between theoretical geometric advantages and practical implementation constraints in multimodal learning.