SETUP: Sentence-level English-To-Uniform Meaning Representation Parser

This paper introduces SETUP, a sentence-level English-to-Uniform Meaning Representation parser that achieves state-of-the-art performance by fine-tuning Abstract Meaning Representation models and leveraging Universal Dependencies converters, thereby enabling the automatic large-scale production of accurate UMR graphs.

The Gist

Here is an explanation of the paper, translated into everyday language with some creative analogies.

The Big Picture: Translating "Human Thought" into "Universal Code"

Imagine you have a massive library of books written in thousands of different languages. Some are famous languages like English or Spanish, but many are rare, spoken by only a few hundred people, with very few dictionaries or computers that understand them.

The authors of this paper are trying to build a universal translator for meaning. They aren't just translating words; they are translating the idea behind the words into a special, standardized diagram called UMR (Uniform Meaning Representation).

Think of UMR as a universal blueprint.

• If you say "The cat chased the mouse" in English, and "La gata persiguió al ratón" in Spanish, the words are different.
• But in the UMR blueprint, both sentences look like the exact same diagram: A cat (actor) + chasing (action) + mouse (receiver).

This is great because once you have this blueprint, you can use it for anything: fixing broken translations, summarizing news, or helping computers understand rare languages.

The Problem: We have the blueprint (UMR), but we don't have a good construction crew (a parser) to automatically draw these blueprints from text. Without a crew, we can't build the library fast enough.

The Mission: Building the "SETUP" Crew

The authors, a team from Amherst College, wanted to build a robot that can read a sentence and instantly draw the correct UMR blueprint. They called their best robot SETUP (Sentence-level English-to-UMR Parser).

They tried two main construction methods to see which one worked best:

Method 1: The "Renovation" Strategy (Fine-Tuning)

Imagine you have a master architect who is already an expert at drawing blueprints for English houses (this is an AMR parser, which is the older, English-only version of UMR).

• The Idea: Instead of hiring a new architect from scratch, they took these existing experts and gave them a crash course on the new UMR rules.
• The Result: They took five different "architects" (AI models like AMRBART, BiBL, etc.) and trained them on UMR data. The best one, BiBL, learned to adapt its English house skills to the new universal blueprint style very quickly. It became the star of the show.

Method 2: The "Assembly" Strategy (UD Conversion)

Imagine you have a different kind of tool that builds a rough skeleton of a house based on grammar rules (this is Universal Dependencies, or UD).

• The Idea: First, use the grammar tool to build a rough, partial skeleton. Then, feed that skeleton to a smart robot (a T5 model) and ask it to "fill in the blanks" and add all the missing details to make a perfect UMR blueprint.
• The Result: This method was surprisingly good! It was like a skilled carpenter who could take a rough frame and finish the house. However, sometimes the robot got confused and added extra brackets or missed a wall, requiring a "clean-up crew" (a post-processing script) to fix the mistakes.

The Twist: The "Minecraft" Problem

The researchers ran into a funny but tricky problem with their data.

• The Old Data (UMR v1.0): This was like a collection of casual, fragmented conversations. "And then he picks it up." "Oops." It was messy but human.
• The New Data (UMR v2.0): A huge chunk of this new data came from people playing Minecraft (a video game where you build things with blocks). The sentences were very specific: "Builder puts down an orange block at X:1 Y:2 Z:-2."

The Result:
The "Renovation" strategy (Method 1) struggled with the Minecraft sentences. Why? Because the original architects were trained on normal English, not on video game coordinates and robot-like dialogue tags like <Architect>.

• When the sentences were normal English, the robots were amazing (scoring 90%+ accuracy).
• When the sentences were about Minecraft blocks, the robots got confused and scored much lower.

The Verdict: What Did They Learn?

1. You can't just copy-paste: You can't just take an English-only tool and expect it to work perfectly on a new, complex system without training.
2. Renovation wins (for now): The best approach was taking existing, powerful English parsers and fine-tuning them. The BiBL model was the MVP, achieving a score of 91 (out of 100) on standard sentences.
3. The "Skeleton" method is a strong contender: The method that builds a rough draft first and then fills it in is also very promising, especially for languages where we don't have good parsers yet.
4. Data matters: If your training data is mostly about building blocks in a video game, your AI will be great at Minecraft but bad at poetry.

Why Does This Matter?

Think of UMR as the DNA of language. If we can automatically read text and extract its DNA (the UMR graph), we can:

• Teach computers to speak and understand rare, endangered languages that have no dictionaries.
• Make translation between totally different languages (like English and Navajo) much more accurate because they are both translating to the same "DNA" first.
• Help search engines understand the intent of a question, not just the keywords.

In short: The authors built a prototype robot (SETUP) that can turn English sentences into universal meaning blueprints. It's not perfect yet (it gets confused by video game lingo), but it's a huge leap forward toward a future where computers truly understand the meaning of words in any language.

Technical Summary

Here is a detailed technical summary of the paper "SETUP: Sentence-level English-To-Uniform Meaning Representation Parser".

1. Problem Statement

Uniform Meaning Representation (UMR) is a graph-based semantic framework designed to capture the core meaning of text across diverse languages, including low-resource ones. While UMR offers significant potential for language documentation, machine translation, and information extraction, its utility is currently limited by the lack of reliable text-to-UMR parsers.

Existing work on automatic UMR parsing is sparse. Prior attempts often rely on converting text to Abstract Meaning Representation (AMR) and then mapping AMR to UMR via rule-based pipelines. However, these approaches struggle with:

• Domain Shift: Performance drops significantly when moving from older, conversational datasets (UMR v1.0) to newer, complex datasets (UMR v2.0) containing specific domains like Minecraft interactions.
• Linguistic Complexity: UMR encodes tense, aspect, modality, scope, and document-level relations, which are harder to capture than standard AMR.
• Low-Resource Barriers: There is a lack of pre-trained parsers for low-resource languages, necessitating a robust English baseline to facilitate transfer learning.

The paper aims to establish strong baselines for sentence-level English text-to-UMR parsing to enable the automatic production of accurate UMR graphs.

2. Methodology

The authors propose and evaluate two primary approaches to text-to-UMR parsing, comparing them against a baseline pipeline.

A. Baseline Approach (Pipeline)

The authors evaluate the existing state-of-the-art pipeline (Chun and Xue, 2024), which follows a Text $\to$ AMR $\to$ UMR trajectory:

1. Text-to-AMR: Uses various pre-trained AMR parsers (AMRBART, SPRING, BiBL, LeakDistill, amrlib).
2. Alignment & Conversion: Uses LEAMR for alignments and Universal Dependencies (UD) trees to generate sentence-level UMRs via rule-based conversion.
3. Neuro-Symbolic Enhancement: Integrates a neuro-symbolic role conversion model (Post et al., 2024) to handle animacy and split-role mapping (e.g., mapping AMR :source to UMR :source, :destination, or :goal).

B. Fine-Tuning Approaches

The paper introduces two novel methods to improve upon the baseline:

1. Direct Fine-Tuning of AMR Parsers (SETUP):

   • The authors take existing high-performing text-to-AMR models (trained on AMR v2.0/v3.0) and fine-tune them directly on UMR data.
   • Models Tested: amrlib (T5), SPRING (BART), BiBL, LeakDistill, and AMRBART.
   • Goal: Adapt the models to UMR-specific structures (e.g., aspect, modality, scope) while retaining semantic knowledge from AMR training.

2. UD-Based Bootstrap Approach:

   • Step 1: Convert sentences to Universal Dependencies (UD) trees using the Stanza pipeline.
   • Step 2: Use a converter (Gamba et al., 2025) to transform UD trees into partial UMR graphs (capturing core semantic info).
   • Step 3: Train a T5 model to convert these partial graphs into complete, gold-standard UMR graphs.

C. Data Strategy

• Datasets: The study utilizes UMR v1.0 (conversational, fragmented) and UMR v2.0 (mixed: Minecraft interactions and standard English news/text).
• Data Splitting: Recognizing that UMR v2.0 is dominated by repetitive Minecraft dialogue (Builder/Architect tags), the authors created specific splits to test generalization. They excluded overlapping sentences used in original AMR training to prevent data leakage.
• Preprocessing: Addressed tokenization mismatches between UD graphs and UMR text (e.g., handling <Architect> tags).

3. Key Contributions

1. SETUP Model: Introduction of SETUP (Sentence-level English-to-UMR Parser), a fine-tuned AMR parser architecture that achieves state-of-the-art performance.
2. Comprehensive Baseline Analysis: A granular evaluation of the existing pipeline approach across UMR v1.0 and v2.0, highlighting the severe performance degradation caused by domain shifts (specifically Minecraft data).
3. Novel Parsing Strategies:
   • Demonstrating that fine-tuning AMR parsers on UMR data is highly effective.
   • Validating a UD-to-Partial-UMR-to-Full-UMR pipeline as a competitive alternative, despite structural challenges.
4. Resource Release: The authors commit to releasing code and checkpoints to facilitate future research in multilingual and low-resource UMR parsing.

4. Results

The models were evaluated using AnCast, SMATCH, and SMATCH++ metrics.

• Baseline Performance: The pipeline approach performed poorly on UMR v2.0 (SMATCH $\approx$ 35.6), significantly worse than on v1.0 (SMATCH $\approx$ 72.2).
  • Reason: The pipeline struggled with Minecraft-specific tokens (coordinates, dialogue tags) and the structural complexity of v2.0.
  • Non-Minecraft Data: On standard English sentences within v2.0, the baseline improved (SMATCH $\approx$ 64.8), confirming that the drop was due to domain mismatch.
• Fine-Tuning Performance (SETUP):
  • BiBL (fine-tuned) emerged as the best performer:
    • AnCast: 84.35
    • SMATCH: 88.82
    • SMATCH++: 90.98
  • AMRBART followed closely as the second-best model.
  • All fine-tuned models significantly outperformed the baseline pipeline.
• UD-Based Approach:
  • The T5 model trained on partial UMRs achieved competitive results (SMATCH $\approx$ 80.6), often surpassing fine-tuned SPRING and LeakDistill.
  • Limitation: The UD approach suffered from structural errors, such as missing or extra parentheses, requiring post-processing scripts to fix graph validity.
• Qualitative Analysis:
  • BiBL excelled at capturing modifiers and complex relations (e.g., :mode expressive, :FR relative-to-builder).
  • UD Approach captured core predicates well but often missed fine-grained nuances (e.g., misaligning modifiers as arguments).
  • Both approaches struggled with non-Minecraft declarative sentences in the test set, indicating a bias toward the Minecraft-heavy training data.

5. Significance and Future Work

• Foundation for Low-Resource Languages: By establishing robust English parsers, this work provides a critical foundation for transferring techniques to low-resource languages where annotated data is scarce.
• Enabling Downstream Applications: Reliable parsers are a prerequisite for using UMR in machine translation, summarization, and information extraction.
• Future Directions:
  • Extending parsing to document-level UMRs.
  • Exploring other AMR architectures (e.g., CLAP, StructBART).
  • Improving generalization to non-Minecraft domains and morphosyntactically complex languages.
  • Adopting newer metrics like AnCast++ for more comprehensive evaluation of temporal and coreference relations.

In conclusion, the paper demonstrates that fine-tuning existing AMR parsers is the most effective current strategy for English text-to-UMR parsing, achieving substantial gains over pipeline baselines and paving the way for scalable, multilingual semantic representation.