Technical Summary: Binary Token-Level Classification with DeBERTa for All-Type MWE Identification

Problem Statement

Multiword Expressions (MWEs) pose a significant challenge for Natural Language Processing (NLP) systems due to their syntactic flexibility and semantic non-compositionality. A primary difficulty lies in detecting discontinuous MWEs, where components are separated by intervening material (e.g., "look...up" in "look the information up"). Traditional approaches often formulate MWE identification as a multi-class sequence labeling task (e.g., BIO or BILOU tagsets), which assigns a single label to each token. However, existing state-of-the-art methods frequently rely on massive Large Language Models (LLMs) with billions of parameters and lack systematic integration of linguistic knowledge. Furthermore, while binary classification has succeeded in span-based Named Entity Recognition (NER), its application to all-type MWE identification remains unexplored.

Methodology

The authors propose a lightweight, comprehensive approach that reformulates MWE identification as a binary token-level classification task, integrating linguistic features and data augmentation.

1. Binary Token-Level Classification Framework

Instead of assigning a single label from a multi-class tagset, the model makes three independent binary predictions for every token $x_i$:

• START: Is this token the beginning of an MWE?
• END: Is this token the end of an MWE?
• INSIDE: Is this token strictly between the start and end of an MWE?

This approach shifts the complexity from quadratic span enumeration ($O(n^2)$) to linear token processing ($O(n)$). During inference, candidate spans are reconstructed by pairing tokens where $p_{start} \geq \tau_{start}$ and $p_{end} \geq \tau_{end}$, provided intermediate tokens satisfy $p_{inside} \geq \tau_{inside}$. Constraints are applied to ensure reconstructed spans match dataset distributions (e.g., 2–6 members, max 13 tokens).

2. Linguistic Feature Integration

The DeBERTa-v3-large model is enhanced with two forms of syntactic knowledge to aid in detecting nominal and discontinuous patterns:

• NP Chunk Features: Tokens are tagged as inside or outside a Noun Phrase (NP) using spaCy. These binary tags are mapped to learnable embeddings and concatenated with contextualized token states to improve the recall of nominal MWEs (e.g., "stock market").
• Dependency Features: Shortest dependency path lengths between token pairs are computed (capped at 5). These serve two purposes:
  1. Encoding: Averaged dependency distances are mapped to 32-dimensional embeddings.
  2. Hard Constraints: During reconstruction, candidates with consecutive members having a dependency distance $>4$ are rejected to filter out implausible discontinuous combinations.

3. Data Augmentation

To address severe class imbalance in training data, two strategies were evaluated:

• Oversampling: Replicating training sentences containing MWEs.
• Lexical Substitution: Replacing non-MWE tokens with semantically similar words.
  The optimal strategy depends on dataset size: oversampling proved superior for the smaller CoAM dataset, while lexical substitution was more effective for the larger STREUSLE dataset.

4. Datasets

• CoAM: A curated dataset of 1,301 English sentences covering all MWE types (NOUN, VERB, MOD/CONN, CLAUSE, OTHER), including 11.3% discontinuous patterns.
• STREUSLE: A larger dataset used for generalization testing, with categories mapped to CoAM-compatible types.

Key Contributions

1. Task Reformulation: The first application of binary token-level classification (START/END/INSIDE) to all-type MWE identification, demonstrating its efficacy over traditional sequence labeling.
2. Efficiency vs. Performance: A DeBERTa-v3-large model (approx. 400M parameters) achieves state-of-the-art results, surpassing Qwen-72B (72B parameters) by 12 F1 points while using 165× fewer parameters.
3. Linguistic Enhancement: Demonstrated that integrating NP chunking and dependency features significantly boosts performance, particularly for discontinuous and nominal MWEs.
4. Generalization: Validated the approach on the STREUSLE dataset, achieving 78.9% F1, confirming that the method captures fundamental MWE structures rather than dataset-specific artifacts.

Experimental Results

The authors conducted a systematic ablation study with 15 model configurations:

• CoAM Performance:

  • The best model (DLT+lo: DeBERTa-large + linguistic features + oversampling) achieved 69.8% F1.
  • This outperforms the previous best (Qwen-72B) by 12 points (57.8% F1).
  • Discontinuous MWEs: The approach doubled the baseline performance for discontinuous patterns, achieving 29.7% F1 compared to Qwen-72B's 17.1%.
  • Type-Specific Recall: Significant improvements were seen across all types, notably CLAUSE (85.7% recall vs. 28.6% for Qwen) and MOD/CONN (84.7% recall).

• STREUSLE Performance:

  • The best model (DBT+la: DeBERTa-base + linguistic features + lexical substitution) achieved 78.9% F1.
  • The binary token-level approach provided a massive improvement over the span-based baseline (19.8% F1), gaining 53.5 points.

• Ablation Insights:

  • Binary token-level classification alone provided a +5.2 F1 boost over span-based prediction.
  • Disentangled attention (DeBERTa) proved superior to standard BERT architectures for MWE detection.
  • Augmentation strategies were dataset-dependent: oversampling worked best for small, high-quality datasets (CoAM), while lexical substitution worked best for larger datasets (STREUSLE).

Significance and Claims

The paper claims that carefully designed smaller models can substantially outperform LLMs on structured NLP tasks. By reformulating the problem as binary token-level classification and integrating explicit linguistic knowledge, the authors demonstrate that resource-constrained deployments can achieve high performance without the computational cost of massive models.

The authors emphasize that their method successfully handles the "all-type" nature of MWEs, a task where previous frameworks often focused only on verbal MWEs or binary idiomaticity. They acknowledge limitations, including the reliance on upstream NLP tools (spaCy) for linguistic features, the difficulty of long-range discontinuous dependencies, and the inability to compute type-specific precision due to the binary formulation (which predicts boundaries but not types). However, the results suggest that this approach captures fundamental aspects of MWE structure, offering a practical and efficient alternative to current state-of-the-art LLM-based solutions.