💻 computer science

Frequency-Aware Mixture-of-Experts Framework for Named Entity Recognition with Adaptive Feedback Mechanism

This paper proposes TriAdaptNER, a frequency-aware mixture-of-experts framework that leverages specialized high- and low-frequency experts guided by an adaptive selector and a differentiable error-driven feedback mechanism to effectively address class imbalance and improve recognition performance for rare entities in Named Entity Recognition tasks.

Original authors: Jie Su, Yanli Chen, Wei Ke, Jinrong Mo, Hanzhou Wu, Zhicheng Dong

Published 2026-09-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jie Su, Yanli Chen, Wei Ke, Jinrong Mo, Hanzhou Wu, Zhicheng Dong

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of human language, computers have become remarkably adept at reading and understanding text. One of their most useful skills is named entity recognition, a process where a machine scans a sentence and points out specific, important things like the names of people, places, organizations, or dates. This ability is the backbone of many modern technologies, from search engines that find relevant news to systems that organize medical records or build maps of knowledge. For years, researchers have taught computers to do this by showing them millions of examples, hoping the machine learns the patterns that distinguish a person's name from a common noun.

However, a persistent problem has long hindered these systems: the real world is not perfectly balanced. In any large collection of text, some types of entities appear constantly, while others are rare. A computer trained on such data often becomes a specialist in the common things, learning to recognize "New York" or "Google" with ease, but stumbling when it encounters a less frequent name or a unique organization. The machine learns to ignore the rare cases because they appear so infrequently that it assumes they are less important. This creates a blind spot where the most interesting or specific information is often lost.

To address this imbalance, a team of researchers from universities in China has developed a new approach called TriAdaptNER. Instead of training a single, massive computer brain to handle every type of name equally, they built a system that acts more like a small, specialized team. Imagine a newsroom where one reporter is an expert on breaking news about major cities and big corporations, while another reporter specializes in obscure local figures and rare historical events. In this new system, different parts of the computer model are assigned these specific roles. One part focuses on the frequent, common entities, while another part is dedicated to the rare and difficult ones.

The researchers found that simply having these two specialists was not enough; they needed a way to decide which expert should handle each specific word in a sentence. To solve this, they added a third component, a kind of manager that looks at the whole sentence and decides which expert should take the lead. This manager considers the context and the likelihood of the entity being common or rare, then blends the opinions of the two experts to make a final decision. The system also includes a clever way for the experts to learn from each other. If the expert for common names makes a mistake on a rare word, the system uses that error to gently push the rare-word expert to pay closer attention, and vice versa. This happens without the experts needing to re-read the text, but rather by adjusting their internal focus during the learning process.

The team tested this framework on five different standard datasets used to measure how well computers recognize names. These datasets cover a wide range of writing styles, from news articles and legal documents to scientific papers about biology. The results showed that the new system performed very well overall, matching or beating other strong methods on most tests. More importantly, when the researchers looked closely at how the system handled different types of names, they saw a clear improvement in recognizing the rare ones. While the system did not become perfect at every single task, it successfully reduced the gap between how well it recognized common names versus rare ones.

The study also revealed that the specific design choices mattered. When the researchers removed the part that managed the experts, or when they stopped the experts from learning from each other's mistakes, the system's performance dropped. This confirmed that the collaboration between the specialized parts was the key to the success. The system worked best when it could dynamically shift its attention, using the right tool for the right job depending on the word it was currently reading.

Despite these successes, the researchers noted that the system still struggles in certain situations. When a name is highly ambiguous and the surrounding text does not provide enough clues, the computer sometimes still guesses the wrong type of entity, often defaulting to a more common guess. This suggests that while the new method is a significant step forward in handling imbalanced data, there is still work to be done to help machines understand the subtle nuances of language that humans grasp so easily. The findings offer a promising path forward for building more robust and fair artificial intelligence systems that do not overlook the rare and unique details of the world.

Technical Summary: Frequency-Aware Mixture-of-Experts Framework for Named Entity Recognition with Adaptive Feedback Mechanism

Problem Statement
Named Entity Recognition (NER) is a fundamental task in Natural Language Processing, yet it faces a persistent challenge in real-world applications: class imbalance. In standard datasets, high-frequency entity types dominate the training signal, causing models to become biased toward common entities while struggling to recognize rare, low-frequency entities reliably. Existing solutions, such as data resampling (e.g., SMOTE) or cost-sensitive learning (e.g., Focal Loss), often introduce noise, require meticulous hyperparameter tuning, or fail to explicitly address the semantic and structural variations between frequent and rare entities. Furthermore, standard ensemble methods often treat entity types uniformly, missing the opportunity to specialize model components based on entity frequency.

Methodology: TriAdaptNER
The authors propose TriAdaptNER, a frequency-aware Mixture-of-Experts (MoE) framework designed to collaboratively address class imbalance through specialized expert roles and adaptive feedback. The architecture consists of three primary functional modules:

Encoder and Structural Enrichment Module:
- Encoder: Utilizes a BERT-based encoder to generate contextual token representations, aggregated via max-pooling from subwords and processed through a BiLSTM to capture sequential dependencies.
- Multi-Granularity Structural Enrichment: This module constructs a word-pair grid using Conditional Layer Normalization (CLN) to model asymmetric token interactions. It enriches these interactions with relative distance and region embeddings. Multi-granularity 2D dilated convolutions (with rates $u=1, 2, 3$ ) capture structural patterns at different scales, which are then pooled into token-level structural summaries. These summaries are fused with contextual representations to form the input for the MoE module.
Frequency-Aware MoE Collaborative Module:
The core innovation involves three heterogeneous branches:
- High-Frequency Expert: A RoBERTa-large model specialized for frequent entity types.
- Low-Frequency Expert: A DeBERTa-v3-large model specialized for rare entity types.
- Selector: A Longformer-large model that captures global context and generates adaptive routing weights.
The framework employs a frequency prior derived from normalized entity-label frequencies in the training set. The selector computes context gates and frequency prior scores to generate routing weights ( $m^R_t, m^D_t, m^L_t$ ) that dynamically fuse the logits of the three branches before CRF decoding.
Adaptive Feedback and Collaborative Optimization:
Instead of explicit iterative re-input, the framework uses a differentiable bidirectional feedback mechanism implemented via loss constraints.
- Expert Supervision: The high-frequency expert is penalized more heavily on low-frequency entities (and vice versa) using a learnable frequency prior.
- Routing Consistency: A loss term ensures the selector's routing decisions align with the entity-label frequency prior.
- Bidirectional Feedback: The prediction error of one expert serves as a corrective signal to strengthen the supervision of the other expert during backpropagation. Specifically, if the high-frequency expert performs poorly on a low-frequency entity, the low-frequency expert receives a stronger corrective signal, and vice versa.
The total objective function combines sequence-level CRF loss, expert supervision loss, routing consistency loss, and bidirectional feedback loss.

Key Contributions

Frequency-Aware MoE Architecture: A novel framework that explicitly integrates entity-frequency characteristics into expert specialization, assigning RoBERTa-large to high-frequency entities and DeBERTa-v3-large to low-frequency entities.
Multi-Granularity Structural Enrichment: A module that converts pairwise token interactions, distance cues, and region cues into token-level structural summaries to aid downstream sequence labeling.
Adaptive Feedback Mechanism: A selector-guided feedback loop where error-driven cross-expert correction is realized through auxiliary loss constraints rather than explicit forward re-input, enabling end-to-end co-optimization.
Comprehensive Evaluation: Extensive experiments across five benchmark datasets (ACE04, ACE05, GENIA, CoNLL03, OntoNotes 5.0) covering both flat and nested NER settings.

Experimental Results

Overall Performance: TriAdaptNER achieves competitive F1 scores against established and recent strong baselines. On CoNLL03, it achieves 93.72% F1, outperforming most listed baselines though slightly trailing the 2026 DEG-NER model (94.46%). On OntoNotes 5.0, it achieves 91.33% F1, outperforming the strongest directly comparable baseline by +0.67 points. On ACE04, it reaches 90.51% F1, surpassing the 2026 DEG-NER model by +1.79 points.
Frequency-Group Analysis: The most significant gains are observed in low-frequency entities. On ACE04, TriAdaptNER improves the F1 score for low-frequency entities from 82.58% (strongest baseline) to 85.73%. On CoNLL03, it improves low-frequency F1 from 85.91% to 88.04%. This confirms the framework's ability to alleviate the performance gap between frequent and rare entities.
Ablation Studies: Removing the frequency prior, routing consistency, or bidirectional feedback loss results in consistent performance drops (e.g., -0.55 F1 on ACE04 without frequency prior), validating the necessity of each component. The removal of the Longformer selector or adaptive routing leads to larger degradation, highlighting the importance of context-aware routing over static aggregation.

Significance and Claims
The paper claims that TriAdaptNER provides a robust solution for class-imbalanced NER by moving beyond uniform treatment of entity types. Its primary significance lies in its targeted improvement of low-frequency entity recognition while maintaining stable performance on frequent entities. The authors modestly note that while the framework is competitive with state-of-the-art systems, it does not uniformly dominate every benchmark in terms of aggregate F1, particularly when compared to models optimized specifically for aggregate scores. Instead, the framework's value is demonstrated through its ability to handle the long-tail distribution of entity types, offering clearer gains where rare entities typically suffer from insufficient training signals. The proposed design aligns contextual encoding, structural enrichment, and frequency-aware routing in a unified architecture that supports error-driven collaborative optimization without explicit feedback loops.

Technical Summary: Frequency-Aware Mixture-of-Experts Framework for Named Entity Recognition with Adaptive Feedback Mechanism

More like this