💬 NLP

Bringing Emerging Architectures to Sequence Labeling in NLP

This paper investigates the applicability of emerging architectures like xLSTMs and structured state-space models to sequence labeling tasks, revealing that their strong performance in simpler settings often fails to generalize across different languages, datasets, and structurally complex tagging problems.

Original authors: Ana Ezquerro, Carlos Gómez-Rodríguez, David Vilares

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ana Ezquerro, Carlos Gómez-Rodríguez, David Vilares

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to read a sentence and put a sticky note on every word to tell you what it is. Is it a noun? A verb? Is it a person's name? A location? This is called Sequence Labeling.

For a long time, the "Gold Standard" robot for this job has been a Transformer (the same brain behind tools like ChatGPT). It's incredibly smart, but it's also heavy, expensive to run, and sometimes overkill for simple tasks.

This paper is like a group of engineers saying, "Hey, let's try some new, different tools to see if we can do this job just as well, or maybe even better, without the heavy machinery." They tested four new types of "robots" (architectures) against the old champion.

Here is a breakdown of their experiment using simple analogies:

1. The Contenders (The New Tools)

Instead of the standard Transformer, they tested four alternative approaches:

The Diffusion Model (The "Denoising Artist"):
- How it works: Imagine you have a picture of a cat, but it's covered in static noise. A diffusion model starts with pure static and slowly removes the noise step-by-step until the cat appears.
- The Experiment: They tried using this for text. Instead of painting a cat, they started with a random mess of labels and slowly "cleaned" the noise until the correct labels appeared.
- The Result: It was like trying to paint a masterpiece by slowly erasing a mess. It worked okay for simple pictures (easy tasks), but when the picture got complex (like a tangled tree structure), the artist got confused and couldn't keep up with the standard robot.
The Adversarial Network (The "Teacher and Student"):
- How it works: Imagine a forger (the Generator) trying to create fake labels, and a detective (the Discriminator) trying to spot the fakes. The forger gets better and better at fooling the detective, and the detective gets better at spotting lies.
- The Experiment: They trained a "forger" to predict labels and a "detective" to check if those labels made sense in the context of the sentence.
- The Result: This was the surprise winner! The forger-detective team learned to spot patterns so well that they often matched or even beat the heavy Transformer, especially on the hardest, most complex puzzles. It proved that "learning by arguing" is a powerful strategy.
The xLSTM (The "Super Memory"):
- How it works: Think of an old-school memory unit (LSTM) that sometimes forgets things if the story gets too long. The xLSTM is a supercharged version with better "forget gates" and the ability to read faster (parallel processing).
- The Result: It was a solid upgrade over the old memory units. It was faster and generally better, but it still couldn't quite catch up to the Transformer's deep understanding of context.
The Mamba/SSM (The "Linear Speedster"):
- How it works: Imagine a runner who usually has to stop and look back at every step they took to decide the next move (quadratic complexity). The Mamba model is a runner who can sprint forward without looking back, making it super fast and efficient.
- The Result: It was fast, but it struggled to understand the long-distance connections in a sentence. It was like a sprinter who runs fast but misses the plot of the story because they aren't looking back.

2. The Test Track (The Challenges)

They didn't just test these robots on simple sentences. They put them through a "Gauntlet" of increasing difficulty:

Level 1: Simple Tagging (Part-of-Speech): "Is this word a noun or a verb?" (Like sorting laundry).
- Outcome: The new tools (especially the Adversarial one) did very well here, almost matching the big Transformers.
Level 2: Named Entities: "Is this word a person, a city, or a drug?" (Like finding names in a phone book).
- Outcome: The Adversarial model shined here too. The Diffusion model struggled with languages that have weird, unbalanced data.
Level 3: Complex Structures (Parsing): "How do these words connect to form a tree or a graph?" (Like untangling a knot of Christmas lights or mapping a subway system).
- Outcome: This is where the new tools mostly failed. The "Speedster" (Mamba) and the "Artist" (Diffusion) couldn't handle the complex web of connections. Only the Adversarial model managed to stay competitive, proving it could handle the "knots" better than the others.

3. The Big Takeaway

The paper concludes with a clear lesson:

Don't throw out the old tools yet: The Transformer is still the king for the hardest, most complex jobs.
But there is hope: The Adversarial approach (the Forger/Detective team) is a game-changer. It's a lighter, faster way to get results that are just as good as the heavy Transformers, even on difficult tasks.
One size doesn't fit all: Just because a new tool works great for generating text (like writing a poem) doesn't mean it will work well for labeling text (like tagging a photo).

In a nutshell: The researchers tried to find a lighter, faster car to replace the heavy SUV (Transformer) for driving on different terrains. They found that while most new cars were too slow or got lost on the winding roads, the Adversarial car was surprisingly agile and could handle the tough terrain just as well as the big SUV.

1. Problem Statement

Sequence Labeling (SL) is a fundamental NLP task where each token in a sequence is assigned a label (e.g., Part-of-Speech tagging, Named Entity Recognition). While Pretrained Transformers (e.g., BERT, XLM-R) currently dominate this field due to their ability to model long-range dependencies, they are computationally expensive and rely on self-attention mechanisms.

Recent advances in language modeling and computer vision have introduced alternative architectures, including:

Recurrent variants: xLSTMs (extended LSTMs with parallelization).
State-Space Models (SSMs): Mamba/SSD (linear-time complexity).
Generative/Discriminative approaches: Diffusion models and Generative Adversarial Networks (GANs).

The Gap: While these architectures show promise in language modeling or simpler tasks, their efficacy in complex, structured sequence labeling tasks (such as constituency parsing, dependency parsing, and graph parsing) across multiple languages remains largely unexplored. Most prior evaluations have been limited to flat tasks like NER or POS tagging. This paper investigates whether these emerging architectures can generalize beyond simple settings and compete with Transformer-based baselines.

2. Methodology

The authors conducted a comprehensive empirical study evaluating four emerging architectural families against a standard Transformer baseline. The evaluation spanned five task categories with varying structural complexity:

POS Tagging: Coarse-grained (Universal Dependencies) and fine-grained (SPMRL, PTB, CTB).
Named Entity Recognition (NER): Multi-lingual datasets (CoNLL-2003, BioNLP, etc.).
Constituent Parsing: Linearized via Relative Encoding (R) and Tetra-tagging (T).
Dependency Parsing: Linearized via Absolute, Bracketing (B, B4, B7), and Hexatagging (H).
Graph Parsing: Handling non-projective, cyclic, and multi-head structures using relative, bracketing, and $k$ -bit encodings.

Architectures Evaluated:

Diffusion Tagging (DiT): Adapts Denoising Diffusion Implicit Models (DDIM). It treats the label sequence as a continuous signal (via bit-conversion) and iteratively denoises a random vector conditioned on the input sentence.
Adversarial Tagging (GaT): Based on GANs. It employs a Generator (encoder-decoder) to predict tags and a Discriminator (BiLSTM) to distinguish between ground-truth and generated tag sequences. The generator is trained to fool the discriminator while minimizing cross-entropy loss.
xLSTM & BixLSTM: Recurrent architectures replacing standard LSTMs. xLSTM introduces parallelizable blocks (mLSTM) and stabilized forget gates (sLSTM). BixLSTM processes sequences bidirectionally.
MAMBA-2 (SSD): A Selective State-Space Model (SSM) designed for linear-time sequence modeling, replacing the Transformer encoder.

Experimental Setup:

Languages: 10+ languages (English, German, Chinese, Basque, French, Hebrew, Hungarian, Korean, Polish, Swedish, Arabic, Bulgarian, Italian, Dutch, Tamil).
Baselines:
- Standard Transformer (XLM-RoBERTa) with a Feed-Forward Network (FFN).
- Span-based and Biaffine parsers for parsing tasks.
Metrics: Accuracy (POS), F1 (NER), UAS/LAS (Dependency), Labeled F1 (Constituency/Graph).

3. Key Contributions

Unified Benchmark: The first large-scale evaluation of Diffusion, Adversarial, xLSTM, and SSM architectures across a spectrum of sequence labeling tasks, ranging from flat tagging to complex graph parsing.
Architecture Adaptation: Novel adaptations of Diffusion and GANs specifically for discriminative token-level classification, including bit-conversion strategies for discrete labels and adversarial loss formulations for sequence validity.
Generalization Analysis: A detailed analysis of how these models perform under varying conditions:
- Label Space Size: How performance degrades as the number of tags increases.
- Structural Complexity: Performance on trees vs. graphs.
- Language Diversity: Cross-lingual robustness.

4. Key Results

The results reveal that the "strong performance" of emerging architectures in language modeling does not always translate to sequence labeling, particularly in complex settings.

Adversarial Tagging (GaT) is the Standout:
- Performance: Consistently rivals or surpasses the Transformer baseline across diverse tasks.
- Complex Tasks: It is the only non-Transformer architecture that maintains competitive performance in complex structured tasks (Dependency and Graph Parsing).
- Mechanism: The adversarial loss encourages the generation of "well-formed" sequences, effectively capturing structural constraints that other models miss. In constituency parsing, it produced valid label sequences for 82.13% of the test set (vs. 80.76% for the baseline).
- Stability: It remains stable even as the label space grows (high entropy), whereas other models degrade significantly.
xLSTM / BixLSTM:
- POS Tagging: BixLSTM (bidirectional) generally outperforms traditional BiLSTMs and is the best non-pretrained encoder, though it still falls short of the Transformer baseline.
- Limitations: Performance drops significantly in parsing tasks, struggling with long-range dependencies compared to bidirectional Transformers.
Diffusion Tagging (DiT):
- Mixed Results: Performs reasonably well on simple tagging (POS/NER) but fails to generalize to complex parsing tasks.
- Weakness: Struggles with highly unbalanced datasets (e.g., Weibo NER) due to reliance on MSE loss. It fails to effectively internalize hierarchical information required for parsing.
State-Space Models (MAMBA-2 / SSD):
- Poor Performance: Consistently trails behind baselines, often performing on par with or worse than simple left-to-right LSTMs.
- Insight: SSMs struggle to capture the specific token-to-token dependencies required for structured prediction in NLP, despite their efficiency in language modeling.
Speed vs. Accuracy:
- While unidirectional encoders (LSTM, xLSTM, Mamba) are fast, their accuracy in parsing is poor.
- The Adversarial tagger offers a speed-accuracy trade-off comparable to the baseline (similar inference time) but with superior structural modeling.

5. Significance and Conclusion

This paper challenges the assumption that emerging architectures (Diffusion, SSMs, GANs) are universally superior or even viable alternatives to Transformers for all NLP tasks.

Main Finding: While Diffusion and SSMs are promising for generative tasks or simple classification, they are currently less suited for complex, structured sequence labeling.
The Exception: Adversarial learning (GANs) has proven to be a powerful paradigm for sequence labeling. By framing the task as a game between a generator and a discriminator, it successfully enforces structural validity and generalizes well across languages and task complexities.
Implication: For tasks requiring strict structural constraints (like parsing), generative adversarial approaches may offer a viable, competitive alternative to heavy Transformer models, potentially reducing computational costs while maintaining high accuracy.

The authors conclude that while Transformers remain the dominant force, Adversarial Tagging represents a significant breakthrough for non-Transformer approaches in structured NLP, whereas other emerging architectures require further refinement before they can replace standard encoders in complex scenarios.