A Scalable Tool for Measuring Manner and Result Verbs in Developmental Language Research

Technical Summary: A Scalable Tool for Measuring Manner and Result Verbs

Problem Statement
The distinction between manner verbs (encoding how an action is performed, e.g., run, scribble) and result verbs (encoding an outcome or change of state, e.g., clean, break) is theoretically significant in developmental language research. It helps explain variations in early verb learning and outcomes for children with language delays. However, studying this distinction at scale is currently hindered by a lack of large-scale, annotated resources. While grammatical annotation (e.g., part-of-speech tagging) is mature, fine-grained semantic categorization of verbs in context remains challenging for computational models, limiting the ability to analyze verb semantics in large developmental corpora.

Methodology
To address the scarcity of gold-standard training data, the authors propose a computational pipeline that leverages Large Language Models (LLMs) to generate annotations, which are then used to train a supervised classifier.

1. Linguistic Framework: The approach is grounded in established linguistic diagnostics for manner and result verbs, including:
   • Syntactic Diagnostics: Object omission (manner verbs often appear without objects; result verbs typically require them) and causative/inchoative alternation (result verbs allow this; manner verbs generally do not).
   • Semantic Diagnostics: Telicity (whether an action has a natural endpoint) and the nature of change (scalar vs. non-scalar).
2. LLM-Based Annotation Pipeline:
   • Data Source: Sentences were drawn from the Manually Annotated Sub-Corpus (MASC) and the InterCorp parallel corpus, covering 4,492 sentences and 2,554 unique verb occurrences.
   • Prompt Engineering: Two distinct prompts were designed for GPT-4o to classify non-stative verbs:
     • Prompt 1 focused on semantic properties (scalar vs. non-scalar changes inherent to the verb root).
     • Prompt 2 focused on sentence structure (object omission and alternation patterns).
   • Sequence Tagging: Rather than binary classification, the task was framed as sequence tagging (similar to POS tagging). The LLMs tagged tokens in sentences, distinguishing between result, manner, stative, and other categories. This allowed for the isolation of non-stative verbs, avoiding misclassification of auxiliary or modal verbs.
3. Model Training:
   • A RoBERTa-base model (125M parameters) was fine-tuned on the LLM-generated annotations.
   • The architecture includes a Byte Pair Encoding tokenizer, the RoBERTa backbone for contextual embeddings, mean pooling for feature selection, and a classification head with label smoothing.
   • The model was trained to predict token-level labels for 17 standard POS tags plus two new semantic tags: result and manner.

Key Contributions

• Scalable Framework: The authors present the first computational framework for identifying manner and result verbs in sentence context, expanding coverage from 151 previously annotated VerbNet classes to 436 classes.
• Annotation Pipeline: They introduce a method for generating training data using linguistically informed LLM prompts in the absence of large-scale gold-standard resources.
• Validated Classifier: They demonstrate that a RoBERTa-based classifier trained on these LLM-generated annotations can reliably distinguish manner and result verbs.
• Resource Release: The code and the newly annotated dataset (covering 436 VerbNet classes) are made publicly available to support future research.

Results
The model was evaluated on three held-out gold-standard datasets:

1. A Linguist verb-root dataset (83 verbs from prior lexical semantics work).
2. A Psycholinguistic verb-root dataset (77 verbs from Horvath et al., 2022).
3. An Expert-annotated dataset (newly created by an expert linguist to cover 346 VerbNet classes).

• Performance: The model trained on semantic prompts (Model 1) consistently outperformed the model trained on syntactic prompts (Model 2).
• Accuracy: Across the three datasets, the best-performing model achieved an average accuracy of up to 89.6%.
  • On the Linguist dataset, Model 1 achieved 94% accuracy.
  • On the Psycholinguistic dataset, Model 1 achieved 90% accuracy.
  • On the Expert-annotated dataset, Model 1 achieved 86% accuracy.
• Observations: The authors note that semantic properties of the verb root appear more critical for this task than sentence structure alone. Discrepancies in the Psycholinguistic dataset (e.g., lower recall for result verbs) were attributed to potential borderline cases or differences in annotation criteria between developmental research and strict linguistic definitions.

Significance and Claims
The authors position this work as a scalable measurement tool rather than a clinical diagnostic instrument.

• Research Utility: The tool enables researchers to measure finer-grained semantic properties of verb vocabularies in large developmental corpora (e.g., CHILDES), potentially clarifying which aspects of early language are associated with later outcomes like Developmental Language Disorder (DLD).
• Modesty and Limitations: The paper explicitly acknowledges limitations, including the reliance on LLMs which may introduce statistical inconsistencies, the exclusion of polysemous verbs and mixed manner/result cases (e.g., guillotine, drown), and the lack of inter-rater reliability due to a scarcity of expert annotators.
• Future Directions: The authors state that further validation is required for borderline cases and downstream developmental applications before the tool can be used for diagnostic decision-making. They view the current system as a step toward richer corpus-based analyses of early verb learning, supporting the hypothesis that semantic composition of input may be more informative than broad lexical measures.