Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification

Technical Summary: Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification

Problem Statement
The paper addresses the challenge of detecting Turkish Light Verb Constructions (LVCs), a specific class of Multiword Expressions (MWEs). Turkish LVCs are difficult for Natural Language Processing (NLP) systems because they often share the same surface form as fully literal verb–object combinations while functioning as a single, partially idiomatic predicate. For instance, the verb ver- (give) appears in both literal transfers (e.g., "gave a pen") and idiomatic LVCs (e.g., "gave inspiration"). The core difficulty lies in distinguishing whether a verb–nominal sequence functions as a unitary idiomatic predicate or a literal argument structure. The authors argue that standard NLP evaluation often fails to capture these human-linguistic distinctions, necessitating a controlled evaluation framework that isolates the literal–idiomatic contrast.

Methodology
The study frames Turkish LVC detection as a binary classification task: identifying whether a sentence contains an idiomatic/LVC predicate ([1]) or a literal verb–argument configuration ([0]).

• Dataset Construction: The authors created a manually curated, controlled diagnostic dataset of 147 sentences. This set includes three balanced conditions (49 items each):
  1. LVC: Idiomatic/light-verb predicates.
  2. NLVC: Literal controls using the same target verbs as the LVC items (to prevent trivial verb-only heuristics).
  3. RANDOM: Out-of-domain negative controls.
     The dataset was validated by two annotators for naturalness and label correctness.
• Models Evaluated:
  • Supervised Baseline: Two variants of BERTurk (32K and 128K vocabularies) fine-tuned with a task-specific binary classification head. Training data was derived from Turkish Universal Dependencies (UD) treebanks, utilizing compound:lvc and compound dependency relations to extract candidate LVCs.
  • Instruction-Tuned LLMs: Three models from different families were evaluated under varying prompting regimes: Llama 3.1 (8B), GPT-OSS (20B), and Qwen 2.5 (14B).
• Experimental Regimes: The LLMs were tested under three conditions:
  1. Zero-shot: Instruction only.
  2. One-shot: Instruction plus one positive (LVC) and one negative (NLVC) example per verb template.
  3. Few-shot: Instruction plus a richer set of labeled examples (multiple positive/negative pairs).
     The supervised BERTurk models were evaluated as a fixed baseline.

Key Results
The experiments reveal significant sensitivity to prompting regimes and model architecture:

• Zero-Shot Performance: LLMs exhibited a strong conservative bias, performing well on negative controls (Random and NLVC) but failing to detect LVCs (recall near 0% for Llama 3.1, very low for others). In contrast, the supervised BERTurk baseline maintained high performance on both negatives and LVCs (F1 ~0.75–0.86), outperforming zero-shot LLMs significantly on the positive class.
• One-Shot Performance: Introducing a single demonstration sharply altered error profiles but introduced strong model-specific biases. Llama 3.1 shifted to over-predicting LVCs (high recall, low precision on negatives), while Qwen 2.5 remained conservative (high precision on negatives, lower recall on LVCs). GPT-OSS-20B showed the most balanced performance.
• Few-Shot Performance: A richer few-shot prompt improved calibration for GPT-OSS-20B and Qwen 2.5, yielding robust overall accuracy (84–86%) and reducing extreme failure modes. However, Llama 3.1 continued to over-predict positives.
• Statistical Analysis: Chi-square tests indicated that while zero-shot differences among LLMs were often insignificant, one-shot and few-shot regimes produced significant cross-model differences in error profiles. The supervised baseline remained competitive with, and in some zero-shot comparisons superior to, the prompted LLMs.

Key Contributions

1. Controlled Diagnostic Dataset: The release of a manually annotated, matched-control dataset (N=147) specifically designed to test the literal–idiomatic distinction in Turkish LVCs, avoiding the noise of standard i.i.d. test sets.
2. Empirical Analysis of Prompt Sensitivity: The study demonstrates that in-context learning does not uniformly improve performance; rather, it shifts decision boundaries and calibration. One-shot prompting can induce severe, model-specific biases (over- or under-prediction), while few-shot prompting offers better calibration.
3. Supervision vs. In-Context Learning: The paper provides evidence that for morphologically rich languages and specific metalinguistic tasks (like LVC detection), a smaller, task-specific supervised encoder (BERTurk) can remain competitive with or exceed larger, instruction-tuned LLMs evaluated via prompting.

Significance and Claims
The authors claim that their findings highlight the substantial prompt sensitivity inherent in Turkish metalinguistic classification. They argue that while prompted LLMs can match or exceed supervised baselines when provided with carefully constructed demonstrations, they are not inherently superior without such calibration.

The paper posits that the success of the supervised baseline stems from direct optimization for the specific label space and exploitation of morphosyntactic cues via treebank-derived supervision, whereas LLMs must infer task definitions from instructions. The authors conclude that LVC detection should be treated as a prompt-sensitive capability and that controlled, condition-wise evaluation is essential to reveal systematic decision-boundary failures that aggregate accuracy metrics might mask. They explicitly state that their results are suggestive regarding the value of task-specific supervision in morphologically rich languages but do not claim that BERT is generally more capable than LLMs; rather, the comparison reflects the difference between in-domain supervised sequence labeling and general-purpose prompting.