Reasoning over Object Descriptions Improves Coreference Resolution in Task-Based Dialogue Systems

1. Problem Definition

The paper addresses the challenge of Coreference Resolution (CR) in Task-Based Dialogue Systems (TDS), specifically within visually grounded environments (e.g., shopping assistants).

• The Core Task: Given a multi-turn dialogue, a scene image, and detailed metadata for objects in that scene, the system must identify which specific object(s) in the scene a user is referring to in their latest utterance (e.g., identifying "the white dress" corresponds to Object ID 52).
• Current Limitations:
  • Data Scarcity & Overfitting: State-of-the-art encoder-based models rely heavily on supervised fine-tuning (SFT) on annotated data. They tend to overfit to dataset-specific artifacts (e.g., domain identifiers, object templates) and fail to generalize to unseen objects or new domains.
  • Multimodal Complexity: Resolving references requires reasoning over dialogue history, visual context, and structured object metadata simultaneously.
  • Rigidity: Traditional models struggle when faced with out-of-distribution (OOD) scenarios or novel object types not seen during training.

2. Methodology

The authors propose a unimodal, test-time reasoning approach using Large Language Models (LLMs) that reframes CR as a generative reasoning task rather than a classification task.

A. Input Representation

Instead of using raw images or complex multimodal encoders, the approach converts all visual and structural data into textual prompts:

1. Dialogue History: Previous turns between user and assistant.
2. Object Metadata: A list of all objects in the scene with attributes (ID, type, color, brand, location).
3. Prompt Engineering: The system uses specific prompting strategies to guide the LLM.

B. Reasoning Strategy (Chain-of-Thought)

The core innovation is the use of Test-Time Reasoning via a structured 5-step Chain-of-Thought (CoT) process embedded in the prompt:

1. Identification: Extract noun phrases and descriptive references from the user's utterance.
2. Attribute Gathering: Retrieve relevant metadata (color, type, location) for the identified references.
3. Mapping: Match the extracted attributes to the specific Object IDs in the scene metadata.
4. Ambiguity Resolution: Use dialogue history to disambiguate if multiple objects match the description.
5. Output: Generate the final Object ID(s) (e.g., <SOM>52<EOM>).

C. Prompt Variations

The study evaluates three prompting strategies:

• Zero-Shot: Task instructions + context only.
• Few-Shot: Zero-shot + 3 examples of input/output pairs.
• Few-Shot + Reasoning: Few-shot examples augmented with the explicit 5-step reasoning trace (CoT).

D. Experimental Setup

• Dataset: SIMMC 2.1 (Situated and Interactive Multimodal Conversations), featuring Fashion and Furniture domains with rich object metadata.
• Models: Evaluated various open-weight LLMs (Llama 3.1/3.3, Qwen 2.5/3, Gemma, Mistral, DeepSeek-R1 distilled).
• Baselines: Compared against encoder-based supervised models (e.g., BARTencoder) and standard fine-tuned LLMs.
• Evaluation: Included in-domain testing, cross-domain generalization (training on Fashion, testing on Furniture, and vice versa), and ablation studies on information access.

3. Key Contributions

1. Test-Time Reasoning for CR: Demonstrated that LLMs can effectively perform coreference resolution by reasoning over structured metadata and dialogue history at inference time, without needing extensive task-specific fine-tuning.
2. Cross-Domain Generalization: Showed that LLMs with reasoning capabilities generalize significantly better to unseen objects and domains compared to encoder-based supervised models, which suffer from severe performance drops in OOD settings.
3. Metadata Representation Insight: Proved that converting structured metadata (e.g., JSON/coordinates) into fluent natural language descriptions significantly improves model performance. LLMs reason better over linguistic inputs than raw structured data.
4. Ablation on Information Sources: Highlighted that while object metadata is crucial, the model's ability to track previously mentioned objects in the dialogue history is the single most critical factor for accurate resolution.

4. Key Results

• Performance Gains:
  • Reasoning vs. Zero-Shot: Adding test-time reasoning improved F1 scores by approximately 10–14 points across models.
  • Precision vs. Recall: Reasoning strategies significantly increased Precision (reducing false positives/over-prediction) at the cost of a slight decrease in Recall. The models became more conservative and accurate in ambiguous contexts.
  • Model Size: Larger models (e.g., Llama 3.3-70B) performed best, but smaller models (e.g., Qwen3-4B) with reasoning strategies achieved competitive results, sometimes outperforming larger models without reasoning.
• Cross-Domain Generalization:
  • Encoder Baselines: BARTencoder achieved 70% F1 in-domain but dropped to **5-6%** in cross-domain settings.
  • LLMs with Reasoning: Maintained robust performance (e.g., ~50-60% F1) when transferring between Fashion and Furniture domains, demonstrating true generalization.
  • Fine-Tuning: Supervised Fine-Tuning (SFT) improved in-domain performance further but did not always yield additional gains over few-shot reasoning for cross-domain transfer; in some cases, reasoning alone was sufficient.
• Metadata Format: Models using Full Natural Language metadata descriptions outperformed those using raw coordinates or JSON structures, confirming that natural language aligns better with LLM pretraining.

5. Significance and Implications

• Shift from Supervision to Reasoning: The paper argues that for complex, grounded dialogue tasks, leveraging the inherent reasoning capabilities of LLMs via prompt engineering is more effective and scalable than training heavy encoder-based models on limited annotated data.
• Robustness in Real-World Scenarios: The approach offers a solution for real-world deployment where labeled data is scarce and objects/scenes are constantly changing (e.g., e-commerce with new inventory).
• Design Guidelines: Provides concrete guidelines for building task-oriented dialogue systems:
  1. Use Chain-of-Thought prompting to align context with metadata.
  2. Convert structured data to Natural Language for better model integration.
  3. Prioritize dialogue history tracking alongside object metadata.
• Future Directions: The authors suggest integrating visual grounding modules directly into the reasoning process and exploring dynamic context selection to further enhance performance.

In summary, this work establishes that structured reasoning over textualized object metadata allows LLMs to overcome the generalization limitations of traditional supervised models, offering a robust, data-efficient path forward for multimodal task-oriented dialogue systems.