💻 computer science

DySem: Uncovering Dynamic Semantic Components via Multilingual Consensus for Calculating Semantic Textual Similarity

DySem is a novel, training-free framework that improves semantic textual similarity by identifying dynamic, sample-specific semantic components within large language models through multilingual consensus, thereby overcoming the limitations of using static, high-dimensional last-layer representations.

Original authors: Kaijie Zheng, Weiqin Wang, Yile Wang, Hui Huang

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kaijie Zheng, Weiqin Wang, Yile Wang, Hui Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, incredibly smart library (a Large Language Model, or LLM) that knows almost everything. You want to ask it: "How similar are these two sentences?"

Usually, when we ask the library this question, it gives us a massive, 4,000-page summary of its thoughts for each sentence. It's like trying to compare two recipes by reading every single page of a 4,000-page encyclopedia about cooking, even though the recipes only need a few ingredients. This is slow, messy, and often includes a lot of irrelevant noise (like the history of the paper the book is printed on) that distracts from the actual meaning.

The paper introduces DySem (Dynamic Semantic Components), a new way to ask the library that is faster, smarter, and cuts out the fluff.

Here is how it works, broken down into simple steps:

1. The Problem: Too Much Noise

Current methods look at the "last page" of the library's notes (the last hidden layer) and read every single word on it.

The Issue: That last page is a mix of everything: the meaning of the sentence, the grammar, the tone, and even random facts the model knows. It's like trying to find the flavor of a soup by tasting the whole pot, including the spoon and the water.
The Size: These notes are huge (thousands of dimensions). It's like carrying a backpack full of bricks just to find a single key.

2. The Solution: The "Universal Translator" Trick

DySem doesn't just read the notes once. It uses a clever trick called Multilingual Consensus.

Imagine you have a sentence: "I am eating less sugar to stay healthy."

Step A: You translate this sentence into 10 different languages (Spanish, French, Japanese, etc.).
Step B: You ask the library to analyze the "thoughts" behind the sentence in all 10 languages.
Step C: You look for the common threads. What parts of the library's brain light up in all 10 languages?
- If a specific "neuron" (a tiny part of the model) fires up in English, Spanish, and French, it's probably about the core meaning (eating, health).
- If a neuron only fires up in English but not the others, it's probably just a quirk of the English language (like a specific grammar rule).

By finding the parts that agree across all languages, DySem filters out the "noise" and isolates the pure semantic core.

3. The "Dynamic" Part: Customizing the Search

Once DySem finds the core "ingredients" (the semantic components) for a sentence, it doesn't use the whole 4,000-page encyclopedia. It creates a custom, short list of just the important pages.

The Magic: When comparing two sentences (e.g., "I eat less sugar" vs. "I eat more fruit"), DySem looks at the short list for Sentence A and the short list for Sentence B.
It combines them into a Joint Semantic Set. This is like taking the unique ingredients from both recipes and only comparing those specific items.
It ignores everything else. If Sentence A talks about "sugar" and Sentence B talks about "fruit," the model ignores the thousands of other pages about "history" or "math" that both sentences might have accidentally included.

4. The Result: Faster and Better

The paper claims that by doing this:

It's Smarter: It finds the true meaning better than previous methods because it ignores the "background noise" of the model's general knowledge.
It's Lighter: Instead of using 4,000 dimensions (pages), it often only needs about 1,000 (or even fewer). It's like comparing two recipes by looking at just the top 10 ingredients instead of the whole book.
No Training Needed: You don't have to teach the library anything new. You just ask it questions in a specific way (using these translations and prompts) to get the answer.

The Analogy Summary

Old Way: Comparing two people by reading their entire life history, including their school grades, grocery lists, and diary entries, to see if they are friends.
DySem Way: Translating their stories into different languages to see what they both agree on, then only comparing those specific shared memories. It's faster, cleaner, and tells you exactly how similar they really are.

The paper proves this works across many different types of AI models (like LLaMA and Qwen) and shows that this "dynamic" approach beats the standard "read everything" approach, even though it uses less data.

Technical Summary: DySem

Problem Statement

Calculating Semantic Textual Similarity (STS) is a foundational task in natural language processing, underpinning applications such as semantic search, clustering, and information retrieval. Current methods utilizing Large Language Models (LLMs) typically rely on extracting the last-layer hidden states with fixed, high-dimensional vectors to compute similarity between text pairs. The authors argue this paradigm suffers from two primary limitations:

Suboptimal Representation: The last hidden layer of LLMs is optimized for next-token prediction, encoding generalized information that entangles core semantics with broader, unrelated knowledge. Consequently, it may not be the most effective component for pure semantic similarity computation.
Redundancy and Noise: The hidden state dimensions of LLMs are often very large (e.g., 4096+), introducing significant redundancy and noise. Using the full dimensionality for similarity calculation incurs unnecessary computational overhead, and prior studies suggest semantic integrity is preserved even after dimensionality reduction.

Existing approaches generally map all texts into a vector using the same fixed last-layer hidden states and full dimensions, failing to leverage the internal dynamic structure of the model or the specific semantic needs of individual text pairs.

Methodology: DySem

The authors propose DySem, a novel, training-free framework that shifts away from static, fixed-dimensional representations toward dynamic, sample-specific semantic dimensions. The method operates in three stages:

1. Extracting Semantic Components via Multilingual Consensus

To identify the core semantic meaning of a text $x$ while filtering out language-specific surface artifacts, DySem employs a multilingual consensus strategy:

Translation: The input text is translated into a predefined set of $m$ target languages.
Prompting: Each language version is processed through the LLM using specific prompt templates (either a standard English prompt or a language-specific prompt) to extract internal state vectors.
Component Selection: Instead of using the final hidden state ( $h^L_n$ ), the method evaluates internal components, specifically the cumulative attention outputs ( $A^L_n$ ) and cumulative FFN outputs ( $F^L_n$ ).
Consensus Identification: A dimension is considered part of the "semantic component set" if it is consistently activated (positive value) across all language versions. The authors define the consensus index set $I(x)$ based on this co-activation.
Dimension Ranking: Within the co-activated set, dimensions are ranked by their mean activation across languages, and the top- $k$ dimensions are selected to form the semantic component set $S(x)$ .

2. Constructing the Joint Semantic Set

For a given text pair $(x, y)$ , DySem constructs a pair-dependent joint semantic set $U(x, y)$ by taking the union of the individual semantic component sets:
$U(x, y) = S(x) \cup S(y)$
This dynamic alignment ensures that the comparison captures dimensions relevant to either text, effectively filtering out irrelevant background noise and focusing on shared semantic concepts.

3. Calculating Similarity over Flexible Dimensions

Similarity is computed using a restricted cosine similarity over the joint semantic set $U(x, y)$ . The vectors are projected onto this shared dimensional subset, and the similarity is calculated as:
$\text{sim}(x, y) = \cos(v_U(x), v_U(y))$
Crucially, the dimensionality of the vectors used for calculation is flexible and determined by the specific input pair, rather than a fixed pre-defined size.

Key Contributions

Dynamic Semantic Extraction: The paper introduces a method to uncover dynamic, sample-specific semantic components within LLMs, moving beyond the static last-layer hidden states.
Multilingual Consensus: It leverages the intuition that core semantic meaning remains invariant across translations to isolate semantic dimensions from language-specific noise.
Flexible Dimensionality: The framework demonstrates that high-quality STS can be achieved using significantly fewer dimensions (often around 1,000 or fewer) compared to the full hidden state size, without model training.
Internal Component Analysis: The work systematically evaluates internal components, finding that cumulative attention outputs ( $A^L_n$ ) often provide superior semantic representations compared to standard hidden states or FFN outputs.

Experimental Results

Extensive experiments were conducted across seven standard STS benchmarks (STS12–STS16, STS-B, SICK-R) using ten different LLMs (including LLaMA2/3.1, Qwen2.5/3, and Phi3 families, both base and instruction-tuned).

Performance: DySem consistently outperformed recent baselines, including PromptEOL, AlignedWVA, and various contrastive prompting methods. For instance, on the Qwen3-8B model, DySem achieved an average score of 80.54, surpassing AlignedWVA by 5.75 points and PromptEOL by 11.63 points.
Efficiency: Despite using significantly lower dimensions (averaging between 658 and 1240 dimensions depending on the model and prompt strategy), DySem achieved superior or competitive results compared to methods using full 4096-dimensional vectors.
Component Validity: Ablation studies confirmed that the selected semantic components significantly outperform random dimension selections. Furthermore, cumulative attention outputs were identified as the most reliable internal representation for STS tasks.
Prompt Sensitivity: The method showed robustness across different prompt strategies (English vs. Language-specific) and vector construction methods (Source vs. Mean), with the combination of language-specific prompts and mean vectors often yielding the best results.

Significance and Claims

The paper claims that DySem provides a new direction for representing texts and evaluating relationships with LLMs by directly mining internal model modules. Its significance lies in:

Training-Free Efficiency: It offers a training-free approach to improve semantic similarity, making it applicable to any pre-trained LLM without fine-tuning.
Dimensional Reduction: It challenges the necessity of using full-dimensional hidden states for semantic tasks, demonstrating that a compact, dynamic subset of dimensions is sufficient and often more effective.
Generalizability: The framework is presented as a general processing step that can be applied before similarity calculation and text retrieval, potentially reducing computational overhead in downstream applications like indexing and retrieval.

The authors acknowledge limitations, noting that the initial encoding cost involves $O(m)$ inference overhead due to required translations and independent forward passes, which may pose challenges for massive-scale, cold-start applications compared to single-pass encoders. However, they argue this cost is concentrated in the offline database building phase, while online queries remain efficient.