Can LLMs Interpret and Leverage Structured Linguistic Representations? A Case Study with AMRs

1. Problem Statement

While Large Language Models (LLMs) have achieved success in various NLP tasks, a critical challenge remains: their ability to interpret and utilize structured semantic representations (such as Abstract Meaning Representations or AMRs) that encode language in concise, abstract forms.

• The Gap: Previous research primarily focused on modifying neural architectures (e.g., adding graph attention mechanisms) to process these structures. There is a lack of systematic evaluation regarding whether off-the-shelf, instruction-tuned LLMs can inherently interpret linearized AMRs to improve performance on downstream tasks.
• The Question: Can LLMs effectively leverage AMRs in prompts to enhance reasoning and understanding, particularly in long-context scenarios, or does the structural abstraction degrade performance?

2. Methodology

The study employs a systematic evaluation pipeline using 8-bit quantized, instruction-tuned versions of three prominent LLMs: Llama 3.1 (8B), Phi-3, and Mistral 7B.

Data and Preprocessing

• AMR Extraction: Text from various datasets was converted into Abstract Meaning Representation (AMR) structures (rooted, directed graphs capturing semantic meaning) using the AMR3-structbart-L and doc-sen-conll-amr-seed42 models via IBM's transition-based neural parser.
• Linearization: The graph-based AMRs were converted into linearized (flattened) text strings to serve as input for the LLMs.
• Datasets:
  • LDC2020T02: Sentence-level AMRs for text regeneration.
  • SQuAD 2.0 & HotpotQA: For single-hop and 2-hop Question Answering (QA).
  • SAMSum: For dialogue summarization (long-context).
  • SNLI & DocNLI: For Natural Language Inference (NLI).

Experimental Setup

The authors evaluated three prompting strategies across zero-shot, 3-shot, and 5-shot settings:

1. Context-Only: Standard prompting with original text.
2. AMR-Augmented: Original text + Linearized AMR in the prompt.
3. AMR-Only: Linearized AMR only (no original text), testing the model's ability to infer meaning directly from the structure.

Evaluation Metrics

• Generation/Summarization: ROUGE-1/2/L, BLEU, and Cosine Similarity (using all-MiniLM-L6-V2 embeddings).
• QA: F1-score.
• NLI: Macro F1-score.

3. Key Contributions

1. Systematic Evaluation: The first comprehensive study evaluating the ability of modern, instruction-tuned LLMs to directly leverage linearized AMRs without architectural modifications.
2. Context-Length Dichotomy: The discovery that AMR integration has divergent effects based on context length: it generally degrades performance in short-context tasks but enhances performance in long-context tasks.
3. AMR Reconstruction Capability: Demonstration that LLMs can effectively reconstruct original text from linearized AMRs, achieving high semantic similarity (up to 81% cosine similarity).
4. Model Scaling Observations: The benefits of AMR augmentation are more pronounced in larger, newer models (e.g., Llama 3.1) compared to smaller or older ones.

4. Key Results

A. Short-Context Tasks (Degradation)

• Single-Hop QA (SQuAD 2.0): Adding AMRs to the prompt reduced performance. For Llama 3.1, the F1-score dropped from 59% (context-only) to 52% (AMR-augmented) in 3-shot settings.
• NLI (SNLI): While Phi-3 showed some zero-shot gains with AMRs, adding few-shot examples favored context-only prompting.
• HotpotQA (2-Hop): Despite being a long-context dataset, the individual documents were small. Stacking multiple small AMRs did not improve performance over context-only prompting, suggesting AMRs are less effective when the context is fragmented into small units.

B. Long-Context Tasks (Improvement)

• Dialogue Summarization (SAMSum): This was the most significant success case.
  • Llama 3.1: Zero-shot cosine similarity improved from 66% (context-only) to 76% (AMR-augmented).
  • Mechanism: AMRs helped the model retain key information and understand long-term dependencies in lengthy dialogues, which raw text often obscures due to token limits or noise.
  • Model Dependency: This improvement was specific to the larger, newer Llama 3.1; Phi-3 and Mistral did not show similar gains.

C. AMR Reconstruction (AMR-to-Text)

• LLMs demonstrated a strong ability to regenerate original text from linearized AMRs.
• Llama 3.1 achieved an 81% cosine similarity with 5-shot prompting, indicating that the models can effectively "read" and interpret the semantic structure of AMRs.

D. Fine-Tuning vs. Prompting

• A Rank-32 LoRA fine-tuning experiment on Llama 3.1 for SAMSum summarization yielded results (75-76%) comparable to few-shot prompting but did not surpass the performance of the few-shot AMR-augmented approach. This suggests that for these tasks, prompt engineering with AMRs is currently more effective than lightweight fine-tuning.

5. Significance and Implications

• Strategic Use of Structured Data: The paper establishes that structured representations like AMRs are not a universal "plug-and-play" solution. They are highly effective for long-context understanding (summarization, complex reasoning) where semantic abstraction helps filter noise and highlight core events, but they introduce noise or complexity in short-context tasks where the raw text is already concise.
• Model Evolution: The results suggest that newer, larger LLMs possess a latent capability to interpret complex semantic structures, a capability that scales with model size and training recency.
• Future Directions: The work opens avenues for integrating other structured representations (e.g., Knowledge Graphs, Discourse Representation Structures) into LLM pipelines, particularly for retrieval-augmented generation (RAG) and long-document analysis, without requiring custom neural architectures.

Conclusion

The study concludes that while LLMs can interpret linearized AMRs, their utility is task-dependent. AMR augmentation is a powerful tool for enhancing long-context tasks like dialogue summarization, significantly boosting performance in state-of-the-art models, but it should be avoided for short-context tasks where it may degrade performance.