Neural Models and Language Model Prompting for the Multidimensional Evaluation of Open-Ended Conversations

1. Problem Statement

The rapid proliferation of generative AI-based dialogue systems has created a critical need for robust, automatic evaluation methods. While human evaluation remains the gold standard, it is costly, time-consuming, and often subjective. Existing automatic metrics (e.g., BLEU, ROUGE) show weak correlations with human judgments in open-domain dialogues.

The specific challenge addressed in this paper is DSTC-12 Track 1, which focuses on the automatic, multidimensional evaluation of open-ended dialogues. The goal is to predict dialogue-level scores for ten distinct dimensions (e.g., Empathy, Trust, Relevance, Proactivity, Overall Quality) that align with human judgments. A key constraint of the challenge was the use of relatively small models (fewer than 13 billion parameters), necessitating efficient strategies beyond massive generative models.

2. Methodology

The authors, representing the ORALIS team, proposed and evaluated four distinct approaches to predict scores for the ten dimensions. They utilized three datasets: the official DSTC-12 dataset (185 dialogues), supplemented by CONTURE (119 dialogues) and FED (125 dialogues) to mitigate data scarcity.

Data Preprocessing

• Dataset Harmonization: Since CONTURE and FED used different rating scales and dimension names, the authors mapped these to DSTC-12 dimensions using heuristics (e.g., mapping "Inquisitive" to "Curiosity").
• Score Rescaling: A linear rescaling function was applied to map scores from external datasets to the specific ranges of DSTC-12 dimensions (e.g., mapping a 3-point scale to a 0–100 range).
• Handling Imbalance: The DSTC-12 dataset was highly imbalanced, with only 29% of dialogues annotated for all ten dimensions.

The Four Evaluation Strategies

1. LM Prompting (Language Models as Judges):

   • Utilized small-to-medium LMs (Deepseek Llama 8B, Deepseek Qwen 7B, Qwen 2.5 7B) as evaluators.
   • Explored various prompting techniques: zero-shot, one-shot, few-shot, and self-consistency.
   • Tested different dialogue context strategies (e.g., last 40%, full conversation, summarized dialogue) to optimize input.
   • The model was prompted to output a score and a brief explanation.

2. Regression Models:

   • Trained encoder-based models to predict scores as continuous real numbers.
   • Architecture: ModernBERT Large encoder (approx. 395M parameters) with a regression head.
   • Advantage: ModernBERT supports an 8K token context window, allowing for the encoding of longer dialogues compared to standard BERT models (512 tokens).
   • Trained using Mean Squared Error (MSE) loss.

3. Classification Models:

   • Trained to predict scores as discrete integer classes.
   • Architecture: Sentence-BERT (SBERT) encoder followed by a Multi-Layer Perceptron (MLP).
   • Scores were rescaled to a discrete range (e.g., [0, 8]) and rounded to the nearest integer.
   • Hyperparameters were tuned via grid search to maximize Spearman correlation.

4. Hybrid Approach:

   • A meta-strategy that selected the best-performing method (Regression or LM Prompting) for each specific dimension based on validation set performance.
   • Exclusion: The classification approach was excluded from the hybrid selection due to its discrete nature and potential mapping errors.

3. Key Contributions

• Comprehensive Comparison: The paper provides a rigorous comparison of LM prompting, regression, and classification approaches under the constraint of small parameter counts (<13B).
• Data Augmentation Strategy: Demonstrated a method for integrating heterogeneous datasets (FED, CONTURE) into the DSTC-12 framework through heuristic mapping and linear rescaling to combat data scarcity.
• Contextual Analysis: Investigated the impact of dialogue context length and summarization on LM prompting performance.
• Diagnostic of Evaluation Challenges: Identified significant inconsistencies between training/validation sets and the test set, particularly regarding score distributions and ranges, which explains performance drops.

4. Results

The systems were evaluated based on the mean absolute Spearman correlation with human judgments on the test set.

• Overall Performance: None of the proposed systems outperformed the baseline (a prompted Llama-3.1-8B-Instruct) in terms of average absolute correlation on the test set (Baseline: 0.17 vs. Best Proposed: 0.15).
• Dimension-Specific Wins: Despite the lower average, the proposed methods outperformed the baseline on most individual dimensions:
  • Classification: Won on 6 dimensions (including Empathy), showing the highest number of winning dimensions.
  • Regression & LM Prompting: Each outperformed the baseline on 5 dimensions.
  • Hybrid: Outperformed on 5 dimensions.
• Validation vs. Test Discrepancy:
  • Overfitting: Regression and Classification models showed high correlations on the validation set (0.4–0.79) but dropped significantly on the test set (often near 0 or negative). This suggests overfitting to the specific score distributions of the training data.
  • Generalization of LM Prompting: Interestingly, LM prompting showed better generalization to the test set for certain dimensions (e.g., Curiosity, Proactivity, Overall), sometimes performing better on the test set than the validation set.
• Negative Correlations: Several dimensions exhibited negative correlations, attributed to conceptual mismatches between how LMs and humans interpret metrics, and inconsistencies in annotation ranges between sets.

5. Significance and Conclusion

• Feasibility of Small Models: The study confirms that small, efficient models (ModernBERT, 7B/8B LMs) can compete with larger systems for specific dialogue evaluation tasks, offering a cost-effective alternative.
• The "Small Model" Paradox: While regression and classification models achieved high validation scores, they failed to generalize, whereas LM prompting (despite modest correlations) proved more robust to distribution shifts in the test set.
• Data Quality Issues: A major finding is that the inconsistency in score ranges and distributions between the training/validation sets and the test set severely hampers model performance. The authors argue that future progress depends more on improving dataset annotation quality and consistency than on model architecture alone.
• Future Directions: The authors suggest dedicated annotation campaigns to harmonize scoring and the use of domain adaptation techniques on larger, external datasets to overcome data scarcity.

In summary, this paper highlights that while automatic evaluation of open-ended conversations remains challenging due to data inconsistencies, a combination of LM prompting and classical neural models offers a viable, multi-faceted approach, with LM prompting showing superior generalization capabilities in the face of distribution shifts.