💬 NLP

RexBERT: Context Specialized Bidirectional Encoders for E-commerce

The paper introduces RexBERT, a family of specialized BERT-style encoders trained on a massive 350-billion-token e-commerce corpus using a three-phase pretraining recipe, which outperforms larger general-purpose models on domain-specific tasks despite having significantly fewer parameters.

Original authors: Rahul Bajaj, Anuj Garg

Published 2026-02-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rahul Bajaj, Anuj Garg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Specialized vs. General Knowledge

Imagine you have two types of librarians.

The General Librarian (General-Purpose AI): They have read almost every book in the world. They know a little bit about everything—history, cooking, science, and movies. They are great at general conversation.
The Specialized Librarian (RexBERT): This librarian has only read books about shopping, products, and retail. They might not know much about quantum physics, but they are an absolute expert at understanding the difference between a "red dress" and a "red shirt," or knowing that a "charger" is a complement to a phone, not a substitute.

The paper argues that for e-commerce (online shopping), the Specialized Librarian is actually better than the General one, even if the General one is much bigger and has read more books overall.

The Problem: The "General" Librarian Gets Confused

Most AI models today are trained on the entire internet. While this makes them smart, they often miss the subtle details of shopping.

The Issue: If you ask a general AI, "Is a battery a substitute for a phone?" it might get confused. In shopping, a battery is a complement (you need both), not a substitute (you don't use one instead of the other).
The Result: General models struggle to tell the difference between products that are similar, products that go together, and products that have nothing to do with each other.

The Solution: RexBERT

The authors built RexBERT, a family of AI models designed specifically for shopping. They didn't just make a bigger model; they made a smarter, more focused one using three main steps:

1. The "Ecom-niverse" (The Specialized Library)

To train their model, they couldn't just use the whole internet. They needed a library full of shopping data.

The Analogy: Imagine sifting through a massive mountain of trash (the whole internet) to find only the gold coins (shopping data).
What they did: They created a massive dataset called Ecom-niverse containing 350 billion words. They used a smart filtering process (like a high-tech sieve) to pull out only content related to fashion, beauty, cars, and electronics from a huge web dataset called FineFineWeb. They even used other AI models to double-check that the data was actually about shopping and not just random ads or news.

2. The "Three-Stage Cooking Recipe" (Training Curriculum)

You can't just throw a model into a shopping dataset immediately; it needs to learn the basics first. The authors used a three-step training recipe:

Phase 1: The General Education. First, they taught the model on a mix of everything (books, code, news, web text). This gave it a solid foundation in how language works.
Phase 2: Stretching the Legs. Shopping pages can be very long (think of a product page with a title, a long description, and a list of 20 features). They taught the model to handle very long texts (up to 8,192 words) so it doesn't have to cut off important details.
Phase 3: The "Shopping Boot Camp" (Annealing). Finally, they slowly shifted the training to focus almost entirely on the Ecom-niverse shopping data. They used a special technique called Guided MLM, which is like a teacher pointing at the most important words in a sentence (like "waterproof" or "size 10") and saying, "Pay extra attention to these!" This helps the model learn the specific language of products.

3. The Result: Small but Mighty

The authors built RexBERT models of different sizes, from very small (17 million parameters) to large (400 million parameters).

The Surprise: Even though their models were 2 to 3 times smaller than the big, famous general models, they performed better on shopping tasks.
The Proof: When tested on tasks like:
- Filling in the blanks: Guessing missing words in a product title.
- Matching products: Deciding if two items are the same, similar, or unrelated.
- General smarts: Even on non-shopping tests (like understanding jokes or grammar), these small shopping experts were surprisingly good, often beating larger general models.

Why This Matters (According to the Paper)

The paper claims that quality data + a good training plan is more important than just making the model huge.

Analogy: It's better to have a small, highly trained expert who knows the specific rules of a game than a giant, untrained crowd that knows a little bit of everything.
The Takeaway: If you want an AI for a specific job (like healthcare, law, or shopping), you don't need to build a massive "god-like" AI. You just need to feed it the right, specialized data and teach it carefully.

Summary

RexBERT is a new AI tool for online shopping. It was built by:

Collecting a massive, clean library of shopping text.
Training it in three steps: General learning $\rightarrow$ Long-text learning $\rightarrow$ Specialized shopping learning.
Proving that a smaller, specialized model can beat larger, general models at understanding products, search queries, and customer intent.

The authors released their data and methods so anyone can build similar "specialized librarians" for other fields, not just shopping.

Technical Summary: RexBERT – Specialized Bidirectional Encoders for E-commerce

Problem Statement

While encoder-only transformers (e.g., BERT, ModernBERT) remain indispensable for retrieval, classification, and ranking due to their favorable cost-latency trade-offs, most general-purpose models are trained on generic corpora with limited coverage of specialized domains. In e-commerce, generic models often fail to capture subtle distinctions between product relationships (e.g., substitutes vs. complements) or recognize fine-grained attributes due to the unique, entity-dense, and semi-structured nature of retail text. Furthermore, the research community has increasingly prioritized decoder-only models for generation, leaving encoder research comparatively underdeveloped despite recent architectural advances.

Methodology

1. Corpus Curation: Ecom-niverse

The authors introduce Ecom-niverse, a 350 billion token corpus specifically curated for e-commerce semantics.

Source: Derived from FineFineWeb, a 4.4 trillion token CommonCrawl-based dataset.
Selection Strategy: Domains were manually inspected and grouped into two categories: those requiring filtering (e.g., Hobby, News) and those with near-complete overlap (e.g., Fashion, Beauty, Automotive).
Pipeline: A modular pipeline was employed involving:
- Domain Selection: Focusing on high-overlap domains to reduce the search space.
- LLM Labeling: Using an instruction-tuned model (Phi-4) to assign binary relevance labels (Relevant vs. Not Relevant) to sampled items.
- QA Auditing: A stronger model (Llama3-70B) audited a subset of labels to refine guidelines and ensure consistency.
- Distillation: LLM labels were distilled into scalable fastText classifiers for high-throughput filtering of the entire domain partitions.
- Cleaning: English-only retention, boilerplate/ad removal, deduplication (MinHash), and profanity filtering.

2. Training Curriculum

The pre-training follows a three-phase curriculum adapted from ModernBERT but tailored to e-commerce statistical properties:

Phase 1: General Pre-training: Trained on 1.7 trillion tokens of diverse open data (web, books, code, technical papers) with 512-token sequences, 30% masking ratio, and high dropout to establish robust token representations.
Phase 2: Context Extension: The sequence length was extended to 8,192 tokens for 250 billion tokens. This phase utilized Rotary Positional Embeddings (RoPE) with NTK-aware scaling and alternating global/local attention to model long product pages and attribute blocks.
Phase 3: Annealed Domain Specialization: The model was specialized on the 350 billion token Ecom-niverse corpus.
- Annealing: Sampling weights were gradually shifted toward e-commerce data to prevent catastrophic forgetting.
- Guided MLM: A targeted masking strategy was introduced where ~5% of sequences per batch prioritized masking domain-relevant spans (entities and attributes) identified by a lightweight mining pipeline. This aimed to strengthen representations of high-value tokens while preserving generalization via standard random masking.

3. Model Architecture

RexBERT adopts a modernized BERT-style encoder based on ModernBERT principles:

Architecture: Biasless layers, pre-normalization, GeGLU activations, and alternating global/local attention to reduce quadratic complexity.
Tokenization: Modern BPE tokenizer (50,368 vocab) derived from OLMo.
Optimization: StableAdamW optimizer with a trapezoidal learning rate schedule (Warmup-Stable-Decay).
Scales: Four model sizes were trained: Micro (17M), Mini (68M), Base (150M), and Large (400M) parameters.

Key Contributions

Ecom-niverse: Release of a 350B-token, domain-specific corpus curated via a reproducible, modular pipeline from open web resources.
Pre-training Recipe: A three-phase training methodology (General $\to$ Context Extension $\to$ Annealed Specialization) incorporating Guided MLM to enhance learning of e-commerce semantics.
RexBERT Models: Training and evaluation of four model scales that demonstrate superior performance on domain-specific tasks compared to larger general-purpose encoders.

Results

Token Classification (Masked Token Recovery)

Evaluated on product titles and descriptions (truncated to 128, 256, 512 tokens) using top-k accuracy ( $k \in \{1, 3, 5\}$ ).

Performance: RexBERT models consistently outperformed general-purpose baselines (ModernBERT, DistilBERT, BERT-mini) across all sizes.
Notable Finding: The RexBERT-mini (68M parameters) surpassed ModernBERT-large (400M+ parameters) in top-1 accuracy on product titles at 128 tokens (65.58% vs. 60.5%).
Scaling: RexBERT-base (150M) achieved higher accuracy than ModernBERT-base across all block sizes and top-k settings.

Semantic Similarity (ESCI Benchmark)

Evaluated on the Amazon ESCI dataset (Shopping Queries) to measure the ability to distinguish Exact, Substitute, Complement, and Irrelevant relationships.

Metric: Spearman's rank correlation between predicted cosine similarities and target relevance scores.
Performance: The RexBERT series (Micro to Large) achieved higher correlation than ModernBERT, Ettin, and EmbeddingGemma models.
Scaling Efficiency: Gains were most pronounced from Micro $\to$ Mini $\to$ Base, suggesting that domain-specific curriculum design captures task-relevant signals more effectively than raw parameter scaling alone.

General Language Understanding (GLUE)

Despite specialization, RexBERT models remained competitive on general NLU tasks (SST-2, QNLI, QQP, MNLI, STS-B).

Transferability: RexBERT-mini outperformed DistilBERT across all five GLUE tasks. RexBERT-micro (17M) consistently outperformed Ettin-17m.
Observation: The models demonstrated that domain specialization via a gradual curriculum does not sacrifice broad language understanding capabilities.

Significance and Claims

The paper posits that high-quality in-domain data combined with a principled training approach provides a stronger foundation for e-commerce applications than indiscriminate scaling alone.

Efficiency: RexBERT models achieve state-of-the-art performance on domain-specific benchmarks with 2–3× fewer parameters than comparable general-purpose encoders.
Reproducibility: The work serves as a case study for building domain-specific encoders from open data. The authors claim the pipeline is modular and applicable to other verticals (healthcare, legal, scientific) by substituting the corpus.
Practical Impact: The results suggest improved representations for attribute extraction, catalog completion, and retrieval ranking, particularly in distinguishing subtle product relationships (substitutes vs. complements) that generic models often conflate. The long-context capability (8k tokens) addresses the need to process full merchant pages without heuristic truncation.