💻 computer science

BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs

This paper introduces BidirLM, a family of high-performance bidirectional encoders created by systematically adapting causal LLMs through a novel prior masking phase, a dual strategy of linear weight merging and data mixture to prevent catastrophic forgetting, and the composition with specialized causal models to seamlessly integrate multimodal capabilities.

Original authors: Nicolas Boizard, Théo Deschamps-Berger, Hippolyte Gisserot-Boukhlef, Céline Hudelot, Pierre Colombo

Published 2026-04-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nicolas Boizard, Théo Deschamps-Berger, Hippolyte Gisserot-Boukhlef, Céline Hudelot, Pierre Colombo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, hyper-specialized chef. This chef is an expert at cooking from left to right. They can write a recipe, bake a cake, or cook a meal step-by-step, but they can only look at the ingredients they've already put in the bowl. They can't peek at the ingredients they will add later to decide what to do now. This is how most modern Large Language Models (LLMs) work: they are causal (forward-looking) generators.

However, there's another type of chef: the bidirectional one. This chef can look at the entire bowl of ingredients at once—what's already there and what's coming next—to understand the context perfectly. This is great for tasks like answering a question about a paragraph or finding a needle in a haystack (search), but building these chefs from scratch is incredibly expensive and time-consuming.

The Problem:
The world is full of amazing "left-to-right" chefs (specialized models for math, coding, safety, vision, and audio). But we've been ignoring them for "understanding" tasks because we thought we needed to build new chefs from scratch just to get that "look at everything" ability.

The Solution: BidirLM
This paper introduces a magical kitchen hack called BidirLM. Instead of hiring a new chef, they take the existing "left-to-right" chefs and teach them how to look backward and forward simultaneously, turning them into powerful "omnidirectional" understanding machines.

Here is how they did it, broken down into simple analogies:

1. The "Masking" Warm-Up (The Critical Step)

Many researchers tried to just flip a switch to make these chefs look backward, but it didn't work well. The paper discovered a secret ingredient: The Masking Phase.

The Analogy: Imagine you are teaching a student who is used to reading a book one page at a time. If you suddenly ask them to read the whole book at once, they get confused.
The Fix: First, you cover up random words in the text (masking) and ask the student to guess them using the whole sentence context. This forces their brain to rewire itself to look at the "future" and "past" simultaneously.
The Result: The paper found that skipping this step is a mistake. You must do this "guessing game" first before teaching them to understand the whole picture.

2. The "Memory Loss" Problem (Catastrophic Forgetting)

When you train a chef to look backward, they often forget how to cook the specific dishes they were originally famous for (like math or coding). This is called Catastrophic Forgetting.

The Analogy: It's like teaching a master carpenter to be a master electrician. If you aren't careful, they might forget how to use a hammer.
The Fix: The authors used a technique called Weight Merging.
- Imagine you have two versions of the same chef: Chef A (the original expert) and Chef B (the new omnidirectional expert).
- Instead of picking one, they took a smoothie blender. They mixed 50% of Chef A's brain with 50% of Chef B's brain.
- The Result: The new chef keeps the original skills (math, coding) and gains the new ability to understand context. They didn't need to retrain from scratch; they just blended the knowledge.

3. The "Swiss Army Knife" (Omnimodal)

The paper didn't stop at text. They wanted to make a chef that understands text, images, and audio all at once.

The Analogy: Usually, to get a chef who can cook, paint, and sing, you'd need three different people.
The Fix: They took their new "Omnidirectional Text Chef" and blended it with a "Vision Chef" (who knows images) and an "Audio Chef" (who knows sound).
The Result: They created BidirLM-Omni. It's a single, compact model that can read a document, look at a picture, and listen to a voice recording, all while understanding the context of all three. It's like a Swiss Army knife that is just as sharp as the individual tools it replaced.

Why This Matters

It's Open Source: They didn't keep this secret. They released the recipe so anyone can do it.
It's Efficient: Instead of spending millions of dollars training new models from scratch, you can take existing, specialized models and "upgrade" them for free (or very cheap) using this blending technique.
It's Better: Their new models beat the current best open-source models on almost every test, from understanding math problems to recognizing images and sounds.

In a Nutshell:
The paper says, "Don't throw away your specialized tools! We found a way to take your existing, high-performance AI models, give them a quick 'context' workout, and blend them with their specialized knowledge to create a super-smart, all-seeing AI that understands text, images, and sound better than anything else out there."

1. Problem Statement

While Causal Large Language Models (LLMs) dominate generative tasks and have spawned a vast ecosystem of specialized variants (for code, math, safety, vision, and audio), representation tasks (such as retrieval, classification, and embedding) remain largely the domain of bidirectional encoders (e.g., BERT-style architectures).

Current efforts to repurpose causal LLMs into bidirectional encoders face three critical limitations:

Lack of Consensus: There is no agreement on the optimal training objectives (e.g., masking vs. contrastive learning) or attention mechanisms.
Catastrophic Forgetting: Adapting models without access to the original pre-training data often leads to a loss of foundational knowledge and performance degradation on general tasks.
Inflexibility: Existing adapted encoders cannot easily integrate with the existing ecosystem of specialized causal models, leaving millions of GPU-hours of specialized knowledge untapped.

2. Methodology

The authors propose a systematic, open-source framework to transform causal decoder LLMs into high-performance bidirectional encoders. The methodology is built on three core pillars:

A. Adaptation Strategy (The "What")

The authors conducted systematic ablations on Gemma3 and Qwen3 families to identify the optimal adaptation pipeline. They defined five variants:

Base: Original causal model.
Bi+Base: Base model with bidirectional attention enabled (no further training).
Bi+MNTP: Bi+Base trained with Masked Next-Token Prediction (MNTP).
Bi+Contrastive: Bi+Base trained with contrastive objectives (InfoNCE).
Bi+MNTP+Contrastive: Sequential adaptation: MNTP followed by contrastive training.

Key Finding: The authors discovered that MNTP is a critical, often omitted prior phase. While contrastive training alone excels at zero-shot embedding quality, it sacrifices performance on tasks requiring full-parameter fine-tuning. The sequential approach (MNTP $\rightarrow$ Contrastive) unlocks the full potential of bidirectional attention, achieving peak performance across both regimes.

B. Scaling Without Original Data (The "How to Scale")

To adapt models without access to the original pre-training corpora (which risks catastrophic forgetting), the authors introduced a dual strategy:

Linear Weight Merging: Merging the adapted weights with the original base model weights. They found that a 50% interpolation ratio optimally balances the new bidirectional capabilities with the original distributional knowledge.
Lightweight Multi-Domain Data Mixture: Incorporating a small fraction (20–30%) of multi-domain data (math, code, multilingual) into the training mix.
Result: Combining weight merging with this data mixture effectively mitigates forgetting, preserving performance on general benchmarks (MIRACL, XNLI, CodeSearchNet) while adapting to new domains.

C. Composability (The "Omnimodal" Extension)

The framework leverages weight merging to compose specialized causal models with the adapted encoder.

Domain Alignment: Merging the text encoder with safety models (e.g., Qwen3Guard).
Modality Alignment: Merging the text encoder with vision (Qwen3-VL) and audio (Qwen3-ASR) specialists.
Omnimodal Construction: The authors constructed BidirLM-Omni-2.5B by merging the textual backbones of three specialized Qwen3 variants (Text, Vision, Audio) and appending their respective frozen modality heads. This allows the model to handle text, image, and audio without retraining the entire pipeline from scratch.

3. Key Contributions

BidirLM Model Family: Release of five open-source encoders:
- BidirLM-270M / 1B (Gemma3-based)
- BidirLM-0.6B / 1.7B (Qwen3-based)
- BidirLM-Omni-2.5B (Text, Vision, Audio)
Methodological Insight: Identification of the MNTP pre-phase as essential for high-quality fine-tuning, correcting the trend of "contrastive-only" adaptation.
Data-Efficient Scaling: A novel recipe for scaling adaptation without original pre-training data using weight merging + multi-domain data mixing.
Ecosystem Integration: A demonstration that specialized causal models can be seamlessly "composed" into a unified encoder via weight merging, bypassing the need for expensive end-to-end retraining.

4. Results

The models were evaluated on a comprehensive suite of benchmarks:

Text Representation (MTEB & XTREME):
- BidirLM variants redefine the Pareto frontier for open-source models.
- BidirLM-270M matches the performance of mmBERT-base (a much larger model) with 10% fewer parameters.
- BidirLM-1.7B outperforms EuroBERT-610m by >1 point on augmented XTREME benchmarks.
- On MTEB (Multilingual v2), BidirLM models achieve top-tier scores, surpassing closed-source competitors like Qwen3-Embedding in specific configurations without using proprietary distillation.
Omnimodal Performance:
- BidirLM-Omni-2.5B sets a new state-of-the-art for open-data omnimodal models.
- It outperforms Nemotron-Omni-3B (4.8B params) across all modalities (Text, Image, Audio) while being nearly half the size.
- It ranks 1st on MIEB (Image) and 3rd on MAEB (Audio), surpassing unimodal specialists that are several times larger.
Specialization Efficiency:
- Merged models reached >93% of peak performance in just 20 training steps (80 samples) for domain specialization (safety), demonstrating extreme sample efficiency.

5. Significance

This work fundamentally shifts the paradigm for creating representation models:

From "Training from Scratch" to "Composing Ecosystems": It proves that the vast ecosystem of specialized causal LLMs (safety, vision, audio) can be efficiently aggregated into a single, powerful bidirectional encoder via weight merging.
Democratization of High-Performance Encoders: By providing a recipe that works without proprietary pre-training data, it enables the community to build state-of-the-art encoders using only open-source causal backbones.
Efficiency: The approach drastically reduces the compute cost of creating multimodal models (requiring only ~250 GPU-hours for the Omni model) compared to training from scratch.

In summary, BidirLM demonstrates that by correctly adapting causal LLMs (via MNTP + Contrastive) and intelligently composing them with specialized variants (via weight merging), one can create compact, high-performance, omnimodal encoders that outperform larger, specialized, or closed-source alternatives.