🤖 AI

TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMs

This paper proposes TTSD-FAR, a parameter-efficient framework that combines Test-Time Self-Distillation with Fisher-Anchored Restoration to enable Large Video-Language Models to robustly adapt to missing or noisy modalities during emotion recognition while preventing performance degradation through stability monitoring and drift correction.

Original authors: Muhammad Haseeb Aslam, Alessandro Koerich, Marco Pedersoli, Ali Etemad, Eric Granger

Published 2026-08-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muhammad Haseeb Aslam, Alessandro Koerich, Marco Pedersoli, Ali Etemad, Eric Granger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the bustling world of artificial intelligence, a new generation of systems has emerged that can watch videos and listen to speech simultaneously, understanding the complex interplay between what we see and what we say. These large video-language models are trained on vast libraries of movies, interviews, and conversations, learning to recognize subtle human emotions by weaving together facial expressions, tone of voice, and spoken words. For these systems to work as intended, they rely on a complete picture: the face, the voice, and the text must all be present. However, the real world is rarely so cooperative. In noisy environments, microphones might fail; in privacy-conscious settings, audio might be silenced; or sensors might malfunction, leaving the system with only a fragment of the story. When a key piece of information is missing, these powerful models often stumble, their understanding collapsing because they cannot fill in the gaps on their own.

This is the specific challenge that researchers set out to solve: how to keep these sophisticated emotion-recognition systems working when parts of the input are missing, without having to retrain the entire massive model from scratch. Retraining such systems is prohibitively expensive, requiring weeks of computing power and specialized hardware, making it impossible to update them for every new situation or missing data scenario. The researchers proposed a clever, lightweight solution that allows the model to adapt itself on the fly, learning to compensate for missing information while it is being used, rather than waiting for a future update.

The core of their approach involves a partnership between two versions of the same model. One version, the teacher, remains frozen and unchanged, having been trained perfectly on complete data where all modalities are present. The other version, the student, is a smaller, adaptable component that operates on the incomplete data arriving in real time. As the system encounters a video with missing audio or text, the student tries to guess the emotional state. To learn how to do this better, it constantly compares its internal understanding against the teacher's understanding of what the full picture should look like. This process, known as self-distillation, guides the student to align its reasoning with the teacher's, effectively teaching the student how to reconstruct the missing semantic information based on the clues that remain.

However, the researchers discovered that simply letting the student keep learning forever leads to trouble. If the student continues to update its parameters indefinitely, it eventually starts to forget what it has already learned, or it begins to drift into making random errors as it chases noise in the data. To prevent this, they introduced a safety mechanism that acts like a vigilant monitor. This system watches the stability of the student's learning process. When the student has found a solid solution and its internal understanding has stabilized, the system locks the student in place, freezing its knowledge to preserve what it has learned. It only unlocks the student again if the incoming data changes significantly enough to suggest that the environment has truly shifted, requiring a new adjustment. This cycle of learning, freezing, and re-evaluating ensures the model remains accurate over long periods of use.

The team tested this method on three different datasets involving human emotion, simulating scenarios where up to half of the input data was missing. They compared their approach against other existing methods that tried to fix the problem by guessing based on confidence levels or by searching for similar past examples. The results showed that those other methods often failed, with performance dropping to near-random levels when the missing information was critical, such as when the text transcript was unavailable. In contrast, the new method maintained high accuracy, staying close to the performance of a model that had access to all the data. Even when half the information was missing, the system successfully adapted, proving that it could recover the lost meaning without needing to be retrained.

Crucially, the study demonstrated that this adaptation comes with very little computational cost. The system only updates a tiny fraction of the model's total parameters, leaving the massive underlying structure untouched. This makes the approach practical for real-world deployment, where speed and efficiency are essential. The researchers found that by carefully monitoring the stability of the learning process, they could prevent the model from degrading over time, a common problem in systems that try to learn continuously. Their work suggests that for large, complex AI systems to function reliably in the messy, unpredictable real world, they need a way to learn from incomplete information without losing their core understanding, a balance achieved by guiding them with a stable reference and knowing exactly when to stop and start learning.

Technical Summary: TTSD-FAR for Missing-Modality Emotion Recognition in LVLMs

Problem Statement

Large Video-Language Models (LVLMs) have achieved state-of-the-art performance in multimodal emotion recognition (ER) by leveraging joint understanding of facial expressions, vocalizations, language, and biosignals. However, their deployment in real-world scenarios is hindered by the inevitability of missing modalities at test time (e.g., audio corruption, sensor failure, or privacy constraints).

The absence of a modality induces a significant distribution shift in the model's internal representation space. Empirical analysis (UMAP visualization and Maximum Mean Discrepancy calculations) reveals that missing inputs not only shift the data distribution but also blur decision boundaries, causing class clusters to collapse. This leads to severe performance degradation (e.g., F1-score dropping from 0.6149 to 0.4785 with 50% text missing).

Existing solutions face critical limitations:

Training-time strategies (e.g., modality dropout, separate models) are computationally prohibitive for billion-parameter LVLMs and cannot adapt to deployment-specific missing patterns.
Test-Time Adaptation (TTA) methods based on entropy minimization or perplexity reduction, which work for discriminative models, fail for autoregressive LVLMs. They often lead to optimization drift or near-random performance because confidence-based objectives are unreliable under modality shifts.
Retrieval-Augmented Generation (RAG) degrades when the observed modality is too weak to retrieve semantically relevant neighbors.
Perpetual adaptation risks overwriting consolidated knowledge (catastrophic forgetting) or accumulating noise once the model has converged, particularly in non-stationary test streams.

Methodology: TTSD-FAR

The paper proposes TTSD-FAR (Test-Time Self-Distillation with Fisher-Anchored Restoration), a parameter-efficient framework designed to adapt LVLMs to arbitrary missing-modality patterns without retraining the base model.

1. Test-Time Self-Distillation (TTSD)

The core adaptation mechanism utilizes a frozen teacher (trained on complete modalities) to guide an adaptive student (operating on masked/partial inputs).

Architecture: The framework employs two Low-Rank Adaptation (LoRA) adapters on a shared pre-trained backbone: a frozen Teacher LoRA ( $\Delta\theta_{tea}$ ) and an adaptive Student LoRA ( $\Delta\theta_{stu}$ ).
Objective: Instead of minimizing output entropy, the student minimizes the feature-level distillation loss ( $L_{distill}$ ) between its intermediate representations and the teacher's representations for the same sample.
$L_{distill} = \sum_{l \in \mathcal{L}} \alpha_l \cdot \| h_{stu}^l(\tilde{x}) - \text{sg}(h_{tea}^l(x)) \|_2^2$
where $\tilde{x}$ is the partial input, $x$ is the complete input, and $\text{sg}$ denotes the stop-gradient operation. This treats missing-modality adaptation as a conditional latent reconstruction problem, encouraging the student to approximate the conditional expectation of the teacher's representation given the observed input.

2. Fisher-Anchored Restoration (FAR)

To address the instability of perpetual adaptation, TTSD-FAR introduces a two-state mechanism (Active vs. Anchored) governed by the geometry of the Fisher information landscape.

Active State (Adaptation): The student LoRA is updated via gradient descent on $L_{distill}$ . The Fisher Diagonal ( $F_t$ ), representing the empirical Fisher information, is updated via an exponential moving average of the squared gradients.
Consolidation (Active $\to$ Anchored): The system monitors the Consolidation Index (CI), which measures the normalized $\ell_1$ rate of change of the Fisher diagonal. When the CI remains below a threshold ( $\tau_{fc}$ ) for $K$ steps, the student is deemed to have converged. The current Fisher diagonal is stored as an anchor ( $F_{anchor}$ ), and the student LoRA is frozen.
Drift Detection (Anchored $\to$ Active): While anchored, the system computes a Fisher Mismatch Index (FMI) for incoming complete-modality samples. FMI measures the ratio of the current gradient squared to the anchored Fisher value. If $FMI > \tau_{fmi}$ , it indicates that the incoming data engages parameter dimensions that were previously unimportant, signaling a genuine distributional shift. Adaptation is reactivated to handle this drift.

Key Contributions

TTSD Framework: A parameter-efficient TTA method tailored for autoregressive LVLMs under missing modalities. It replaces unreliable entropy/perplexity objectives with conditional latent representation alignment, effectively recovering semantic information lost due to missing modalities.
FAR Mechanism: A novel two-state stopping and reactivation mechanism. By monitoring the Fisher information diagonal, FAR detects convergence to prevent noise accumulation and detects distributional drift to prevent catastrophic forgetting, ensuring stability over long adaptation horizons.
Empirical Validation: Extensive experiments on three challenging datasets (MELD, DFEW, BAH) with missing ratios ranging from 0% to 50% across text and vision modalities.

Results

Experiments demonstrate that TTSD-FAR consistently outperforms state-of-the-art baselines (TENT, EATA, RAG, Perplexity-based generation) under missing-modality conditions:

Robustness: While entropy-based methods collapse to near-random performance and RAG degrades as the dominant modality is removed, TTSD-FAR maintains performance close to the full-modality baseline.
Stability: In long-horizon adaptation scenarios, baselines without restoration mechanisms progressively degrade due to drift. TTSD-FAR remains consistent, validating the efficacy of the Fisher-Anchored Restoration.
Efficiency: The method updates only a negligible number of parameters (approx. 0.629% of the 7B Video-LLaVA model parameters) and incurs minimal computational overhead, requiring only two forward passes for complete-modality samples and one for missing-modality samples.

Significance and Claims

The paper positions TTSD-FAR as a necessary solution for the "fundamental challenge" of missing modalities in LVLM deployment, a problem previously addressed only by impractical retraining or ineffective TTA methods.

Theoretical Insight: The authors claim that missing modalities induce a specific type of distribution shift where confidence-based supervision (entropy) fails because autoregressive models can generate overconfident, semantically incorrect outputs. TTSD-FAR addresses this by shifting the supervision signal from output confidence to feature-level alignment with a reliable teacher.
Practical Impact: The framework enables LVLMs to operate robustly in real-world, non-stationary environments where sensor data is incomplete, without requiring source data or expensive retraining.
Modesty: The authors acknowledge that the method requires a pre-trained model capable of handling multiple modalities and that the computational cost is slightly higher for complete-modality samples (due to the dual forward pass), though this is negligible for the missing-modality cases which constitute the primary challenge.

The work concludes that by coupling self-distillation with Fisher-based drift detection, it is possible to achieve stable, effective adaptation for LVLMs in the face of missing modalities, preserving learned representations while retaining the capacity to adapt to genuine distributional shifts.