Impact of enriched meaning representations for language generation in dialogue tasks: A comprehensive exploration of the relevance of tasks, corpora and metrics

1. Problem Statement

Conversational systems require Natural Language Generation (NLG) engines that can produce diverse, fluent, and semantically accurate sentences based on structured inputs known as Meaning Representations (MRs). While Large Language Models (LLMs) have advanced NLP, they face challenges regarding hallucinations, lack of reasoning, and opacity. Traditional end-to-end dialogue systems often struggle with coherence and interpretability.

The core problem addressed is whether enriching the input to an NLG generator with a task demonstrator (a specific MR–sentence pair extracted from the dataset) improves generation quality compared to using only the raw MR. Furthermore, the paper investigates how this impact varies across different domains, corpus characteristics (size, variability), and evaluation metrics (lexical vs. semantic).

2. Methodology

2.1 Experimental Setup

• Model: The authors utilized GPT-2 Medium, chosen for its computational accessibility and proven suitability for fine-tuning and prompt-based learning, rather than massive LLMs like GPT-4.
• Datasets: Four distinct dialogue-oriented corpora were selected to ensure diversity in domain, size, lexicon, and acquisition process:
  1. E2E: Single-domain (restaurants), crowdsourced, low MR variability.
  2. ViGGO: Single-domain (video games), crowdsourced, high DA variability.
  3. MultiWOZ: Multi-domain (7 domains), Wizard-of-Oz (WOZ) + crowdsourced, large scale, high complexity.
  4. EMPATHIC: Single-domain (health coaching), WOZ, small scale, high semantic complexity.
• Meaning Representations (MRs): The study compared a Baseline (standard MR with Dialogue Act and slot-value pairs) against three Enriched Prompt representations:
  • Prompt 1: Input MR + Demonstrator with the same Dialogue Act (DA).
  • Prompt 2: Input MR + Demonstrator with the same DA and same number of attributes.
  • Prompt 3: Input MR + Demonstrator with the same DA and exact same attributes (most specific).
  • Note: Demonstrators were selected to minimize the number of unique examples in training while maximizing specificity.

2.2 Evaluation Metrics

The study employed a comprehensive suite of metrics to evaluate different linguistic aspects:

• Reference-Driven Metrics:
  • BLEU: Lexical overlap (n-gram).
  • BLEURT: Semantic similarity trained on human ratings.
  • LaBSE: Language-agnostic embedding similarity.
• Reference-Free (Source-Based) Metrics:
  • Slot Accuracy: Percentage of input slot values correctly present in the output.
  • Dialogue Act Accuracy (DAC): Classification accuracy of the generated sentence's communicative intent (DA).

2.3 Training Procedure

Models were fine-tuned using 5-fold cross-validation over 5 epochs. During inference, five distinct outputs were generated per MR to analyze score distributions and variability.

3. Key Contributions

1. First Comparative Study on Enriched MRs: This is the first study to systematically analyze the impact of task demonstrators on NLG quality across multiple domains, corpus sizes, and evaluation metrics.
2. Prompt Design Innovation: The proposal of three levels of prompt specificity (DA-only, DA+Count, DA+Exact Attributes) and the finding that Prompt 3 (most specific) generally yields the best results, particularly in complex tasks.
3. Metric Analysis: A critical evaluation showing that semantic metrics (BLEURT, LaBSE) capture generation quality more accurately than lexical metrics (BLEU), and that human-trained metrics (BLEURT) detect subtle omissions better than embedding-based metrics (LaBSE).
4. Corpus Characteristic Insights: Demonstration that dataset features (number of references, number of attributes) significantly influence metric scores and the effectiveness of enriched inputs.
5. Zero-Shot Capabilities: Evidence that enriched inputs activate zero-shot capabilities in medium-sized models (GPT-2) even before fine-tuning.

4. Key Results

4.1 Impact of Enriched Representations

• Complexity & Size: Enriched inputs (Prompts) are most effective for small datasets with high variability and complex tasks (e.g., EMPATHIC). In these cases, Prompt 3 significantly outperformed the Baseline.
• Large Datasets: For large-scale datasets like MultiWOZ, the Baseline often performed comparably or better, suggesting that massive datasets may not require specific demonstrators, or that generic demonstrators act as distractors.
• Zero-Shot Performance: All Prompt representations showed higher scores than the Baseline at Epoch 0 (pre-training), proving that a single task demonstrator can immediately improve generation quality in zero-shot settings.

4.2 Metric Performance

• Lexical vs. Semantic: BLEU scores were consistently low (<0.3) and failed to correlate with human judgment or semantic accuracy. **BLEURT** and **LaBSE** provided more reliable assessments (scores >0.5).
• Semantic Nuance: BLEURT (trained on human ratings) was superior to LaBSE in detecting omissions and subtle semantic errors.
• Robustness: Slot Accuracy and Dialogue Act Accuracy were exceptionally high (>90% for MultiWOZ/EMPATHIC), indicating that the models are robust in preserving semantic content and communicative intent, regardless of the metric used for surface-level evaluation.

4.3 Influence of Corpus Characteristics

• Number of References: In datasets with high output variability (MultiWOZ, EMPATHIC), increasing the number of references per MR led to lower BLEU/LaBSE scores because the generated sentence is less likely to match all diverse references. In low-variability datasets (E2E), the number of references had little impact.
• Number of Attributes: The benefit of enriched representations (Prompts) increased as the number of attributes in the input MR increased.
• Artificial Inflation: In small datasets (EMPATHIC), perfect scores were observed for MRs with many attributes but only one reference, caused by the model simply copying the demonstrator sentence (which was identical to the reference).

5. Significance and Conclusion

This paper provides critical insights for the development of robust dialogue systems:

• Practical Application: For small, complex, or highly variable domains, enriching inputs with specific task demonstrators (Prompt 3) is a highly effective strategy to enhance generation quality without requiring massive LLMs.
• Evaluation Best Practices: The study argues against relying solely on BLEU. It advocates for a multi-metric approach where semantic metrics (especially those trained on human ratings like BLEURT) and reference-free metrics (Slot/DAC Accuracy) are essential for accurate evaluation.
• Model Adaptability: Even medium-sized models like GPT-2 demonstrate rapid adaptability to new tasks and robustness in semantic intent when guided by enriched representations.

In summary, the work establishes that the "one-size-fits-all" approach to NLG evaluation and input representation is insufficient; the optimal strategy depends heavily on the specific characteristics of the corpus and the complexity of the task.