Technical Summary: STEB – Style Text Embedding Benchmark

Problem Statement

While semantic text embeddings are rigorously evaluated through standardized benchmarks like the Massive Text Embedding Benchmark (MTEB), the evaluation of style embeddings remains fragmented. Current research relies on disparate tasks, datasets, and evaluation protocols (e.g., varying preprocessing, text length handling, and document-to-embedding ratios). This lack of standardization makes cross-work comparisons unreliable and hinders the quantification of progress in style representation learning. Furthermore, leading general-purpose semantic models, optimized for tasks like semantic similarity and natural language inference, often fail to capture stylistic nuances, yet there is no dedicated benchmark to expose this gap.

Methodology

The authors introduce STEB (Style Text Embedding Benchmark), an open-source framework designed to standardize the evaluation of style embeddings.

Benchmark Composition

STEB comprises 96 datasets across 7 languages, organized into 5 evaluation tasks:

1. Pair Classification: Determines if two texts share a label (e.g., same author). Evaluated via AUROC using both "All-to-All" and "Predefined" pair settings to control for confounding variables like topic.
2. Clustering: Assesses the ability to cluster texts by style-relevant labels (authorship, register, LM provenance) using mini-batch k-means and V-measure.
3. Authorship Retrieval (AR): Identifies other texts by a specific author within a candidate pool, reported via Mean Reciprocal Rank (MRR).
4. Order Alignment: Evaluates if an embedding ranks a set of texts in the same stylistic order as a reference set. Crucially, this includes a distractor variant where an unordered set contains a text with the same topic but unrelated style, forcing models to ignore semantic similarity.
5. Probing: Assesses encoded linguistic features by training linear classifiers on frozen representations, excluding features that correlate with semantics.

Evaluation Scores

STEB reports two complementary aggregate scores:

• Operational Score: A macro-averaged score across datasets and tasks as currently constructed by the field (heavily weighted toward authorship attribution), without committing to a specific theoretical definition of style.
• Definitional Score: A re-weighted score based on the style definition proposed by Wegmann et al. (2026), evaluating whether embeddings encode linguistic features, capture stylistic patterns of various objects (dialect, genre, idiolect), and remain sensitive to style over content.

Experimental Setup

The benchmark evaluates 40 models across four categories:

• Style Embeddings: Contrastively trained models (e.g., LUAR, STAR, CISR, StyleDistance) designed to capture authorship or style.
• Semantic Embeddings: General-purpose models (e.g., Qwen3-Embedding-8B, E5, BGE) optimized for semantic retrieval.
• Masked Language Models (MLMs): Pre-trained transformers (e.g., DeBERTa, RoBERTa) without style-specific fine-tuning.
• Causal Language Models (CLMs): Autoregressive models (e.g., GPT-2 XL, OPT) used to assess implicit style capture.
• Non-Neural Baselines: Hand-crafted features including function word frequencies, TF-IDF n-grams, and stylometric features (LFTK, NeuroBiber).

Protocol: STEB preserves ecological validity by evaluating on full text lengths. For models with fixed context windows, a "chunk-and-pool" strategy is employed. All stochastic steps use fixed random seeds for reproducibility.

Key Results

1. No Universal Winner

No single model dominates across all tasks or definitions. Performance is highly dependent on the downstream goal:

• Authorship Retrieval (AR): Models like LUAR-CRUD and STAR excel, likely benefiting from content-entangled features where author and topic correlate.
• Content Independence: Models trained with hard negatives to isolate style from content (e.g., CISR, StyleDistance) perform best on tasks requiring the disentanglement of style from semantics, such as the distractor variant of Order Alignment.
• Linguistic Feature Sensitivity: RoBERTa-large leads in probing tasks, suggesting that style-specific fine-tuning can sometimes degrade sensitivity to foundational linguistic features.

2. Semantic Models Fail on Style

General-purpose semantic embeddings (e.g., Qwen3-Embedding-8B, E5) consistently underperform specialized style embeddings on stylistic tasks. Notably, top-performing MTEB models rank poorly on STEB, highlighting that semantic objectives do not inherently capture stylistic signals.

3. MLMs are Strong Baselines

Surprisingly, off-the-shelf MLMs (specifically DeBERTa-v3-large and RoBERTa) outperform all causal language models and many specialized style embeddings on authorship verification and retrieval, despite having no embedding-specific fine-tuning. This suggests that pre-training objectives in modern MLMs implicitly capture significant stylistic information.

4. Non-Neural Models Remain Competitive

Traditional non-neural approaches, particularly TF-IDF weighted n-grams and stylometric features, perform surprisingly well, nearly matching sophisticated sentence embedders on probing and predefined classification tasks. However, they lag in clustering and retrieval tasks requiring richer representations.

5. Multilingual Challenges

Multilingual style embeddings (e.g., MSR, mStyleDistance) underperform their English counterparts on multilingual datasets. The results suggest current multilingual training strategies are not yet effective for style representation, though models trained to separate content from style show slightly better generalizability.

6. Efficiency vs. Generative Models

The paper demonstrates that embedding-based approaches (e.g., LUAR-CRUD) significantly outperform generative LLMs (e.g., GPT-5.2) on authorship retrieval tasks while using orders of magnitude fewer FLOPs (approx. 750x less), reinforcing the relevance of text representations for efficiency-critical applications.

Significance and Claims

The authors position STEB as a "yardstick" for measuring progress in style representation learning. The primary significance of the work lies in:

• Standardization: Providing a unified, open-source framework to replace fragmented evaluation practices, enabling reliable cross-work comparisons.
• Gap Identification: Demonstrating that the current dominance of semantic embeddings in MTEB does not translate to stylistic tasks, necessitating dedicated benchmarks.
• Protocol Importance: Showing that evaluation protocols (e.g., handling long texts via chunking vs. truncation) drastically alter model rankings, reinforcing the need for standardized procedures.
• Future Directions: Highlighting that while specialized models lead in specific niches, the field lacks a universally superior model. The authors suggest that future research should focus on improving content-independence, enhancing multilingual capabilities, and exploring joint training objectives that preserve linguistic feature sensitivity while capturing style.

The paper concludes by releasing the code and datasets to encourage community contributions, aiming to lower the cost of measuring progress and foster the development of more style-aware NLP systems.