Technical Summary: Do Visual Features Improve Other-Initiated Repair Detection? A Dyadic Multimodal Approach

Problem Statement

Other-Initiated Repair (OIR) is a fundamental conversational mechanism where a recipient signals a problem in speaking, hearing, or understanding, prompting the original speaker to resolve it. While accurate OIR detection is critical for conversational agents to maintain engagement and prevent interactional failure, existing computational approaches rely almost exclusively on linguistic (text) and acoustic (audio) modalities.

Conversation Analysis (CA) literature establishes that OIR initiation is inherently multimodal, involving coordinated non-verbal signals such as gaze shifts, facial expressions (e.g., eyebrow raises, furrows), head movements, body postures, and gestures. However, a significant gap exists between interactional theory and computational modeling, as no prior study has systematically investigated how visual behaviors contribute to automatic OIR recognition. Furthermore, identical verbal repair expressions can signal different repair types depending on accompanying non-verbal behaviors, necessitating visual integration for robust and interpretable detection.

Methodology

Data and Annotation

The study evaluates a novel multimodal model on two distinct corpora:

1. NOXI: A French, screen-mediated dyadic corpus (role-play dialogue) with frontal camera views.
2. CABB-S: A Dutch, face-to-face, task-oriented corpus (object-matching) with semi-frontal and overview camera views.

Both corpora are annotated following the CA framework of Dingemanse and Enfield, categorizing repair initiations into three types: open requests, restricted requests, and restricted offers. The data is highly imbalanced, with OIR instances constituting a small fraction of total segments.

Model Architecture

The authors propose TriModal, a hierarchical multimodal classifier that processes dialogue segments (target segment + context) across three branches:

1. Text Branch: Utilizes pretrained encoders (RoBERTa/CamemBERT for French, RoBERTa for Dutch) augmented with handcrafted linguistic features.
2. Audio Branch: Uses the Whisper encoder for acoustic representations, augmented with handcrafted prosodic features.
3. Visual Branch (Novel Contribution): Extracts handcrafted visual features grounded in CA literature using OpenFace 2.0 (facial Action Units, head pose, gaze) and MediaPipe Holistic (upper-body and hand landmarks).
   • Feature Sets: The model captures features from the current speaker, the recipient (encoding pre-floor trouble signals), and interactive dyadic patterns (e.g., mutual gaze).
   • Temporal Aggregation: To handle dynamic signals without requiring frame-level sequences, features are aggregated over three equal windows (early, mid, late) of the segment using mean intensity, proportion of frames above threshold, or binary presence flags.
   • Normalization: Continuous features are z-normalized against each speaker's per-session baseline to reflect individual deviations rather than corpus-wide norms.

The model employs a hierarchical classifier: first predicting a binary label (OIR vs. Non-OIR), and if positive, classifying the specific repair type.

Experimental Setup

The study compares four configurations:

• Zero-shot MLLM: Qwen2.5-Omni-7B (end-to-end reasoning without fine-tuning).
• Baseline (Text+Audio): A multimodal model using only text and audio (from prior work).
• UniVis: A unimodal model using only handcrafted visual features.
• TriModal: The proposed full multimodal model.

Evaluation uses 5-fold cross-validation at the session level, reporting Precision, Recall, and Macro F1-score to address class imbalance.

Key Contributions

1. First Computational Study on Visual OIR: Introduces the first multimodal model specifically designed to integrate visual behaviors for OIR detection and classification.
2. CA-Grounded Visual Features: Develops a structured feature set capturing interactionally motivated non-verbal cues (gaze, facial displays, body holds, gestures) derived from Conversation Analysis.
3. Cross-Linguistic and Cross-Context Evaluation: Validates the approach across two languages (French, Dutch) and interaction settings (screen-mediated vs. face-to-face task-oriented).
4. Feature-Level Analysis: Provides insights into how visual behaviors contribute to detection and classification, including ablation studies, gradient saliency, SHAP importance, and cross-modal co-activation analysis.

Results

Performance Improvements

• Detection: Incorporating visual features consistently improved performance over text-audio baselines.
  • NOXI: +4.1 percentage points (pp) in Macro F1.
  • CABB-S: +12.9 pp in Macro F1.
  • The visual-only model (UniVis) achieved performance comparable to the text-audio baseline, confirming that visual cues carry meaningful information independently, though fusion yields the best results.
• Classification: Visual features showed varying impacts on repair-type classification.
  • NOXI: +13.2 pp improvement, suggesting visual cues help disambiguate lexically similar repair types in screen-mediated dialogue.
  • CABB-S: +2.4 pp improvement, as the task-oriented setting produces lexically explicit repairs already well-captured by text/audio.
• Baseline Comparison: All task-specific models significantly outperformed the zero-shot MLLM (Qwen2.5-Omni-7B), indicating that general multimodal reasoning is insufficient for fine-grained OIR detection without task-specific training.

Feature Importance and Interaction

• Modality Ablation: In NOXI, text content was the primary signal, followed by handcrafted linguistic, prosodic, and visual features. In CABB-S, prosodic features contributed more than linguistic handcrafted features, while visual features remained significant.
• Temporal Dynamics:
  • NOXI: Early-phase visual features (e.g., eyebrow furrowing, gaze fixation, lip depression) were most informative, aligning with CA findings that non-verbal signals often precede verbal repair.
  • CABB-S: Late-phase body and head movements (e.g., body torque, leaning, head tilting) were dominant, suggesting a shift in temporal patterns for task-oriented interactions.
• Cross-Modal Co-activation:
  • In NOXI, linguistic and visual cues were highly co-activated, but "hold release" (body movement) showed lower co-activation, indicating it operates as an independent cue.
  • In CABB-S, linguistic, prosodic, and visual cues were almost perfectly co-activated (scores near 1.0), suggesting a tightly coordinated multimodal signal in task-oriented settings.

Significance and Claims

The paper claims that visual features consistently improve OIR detection and classification over text and audio baselines, bridging the gap between interactional theory and computational modeling. The study demonstrates that:

1. Context Matters: The utility of visual features varies by interaction setting. Visual cues are particularly crucial for disambiguating repair types in screen-mediated dialogues (NOXI) and for improving detection accuracy in task-oriented, face-to-face interactions (CABB-S).
2. Interpretability: By using CA-grounded handcrafted features rather than end-to-end visual reasoning, the model provides interpretable characterizations of repair behavior, revealing specific temporal and modality-specific patterns (e.g., early facial cues vs. late body cues).
3. Limitations of General Models: Zero-shot multimodal large language models fail to capture the nuanced, fine-grained interactional phenomena of OIR without task-specific fine-tuning.

The authors conclude that integrating CA-grounded visual features offers both performance gains and a deeper understanding of the multimodal nature of conversational repair, laying the groundwork for more communicatively competent conversational agents. Future work is proposed to evaluate the framework on larger corpora and investigate real-time deployment.