LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations

1. Problem Definition

The paper addresses a critical gap in Incomplete Multimodal Learning (IML): the reliance on the unrealistic assumption that complete data (all modalities present) is available during the training phase.

• The Reality Gap: In real-world scenarios (e.g., sensor failures, privacy constraints, data corruption), training data often suffers from systemic missingness. Existing methods typically use "God's eye view" training (complete data) to generate reconstruction targets or distill priors, which fails when complete data is unavailable during training.
• The Challenge: The authors define the problem of Training-Time Incomplete Observations, where the model must learn robust representations using only partial modalities ($X_{obs}$) and a missingness mask ($m$) during both training and inference, without access to ground-truth complete data for reconstruction or distillation.
• Target Task: The framework is instantiated for Action Quality Assessment (AQA), a task requiring long-term temporal understanding and fine-grained semantic analysis (e.g., sports scoring, medical evaluation), where missing modalities (RGB, Audio, Optical Flow) are common.

2. Methodology: LIMSSR Framework

The authors propose LIMSSR (LLM-Driven Incomplete Multimodal Sequence-to-Score Reasoning), which reformulates IML as a conditional sequence reasoning task rather than a reconstruction task. It leverages the world knowledge and reasoning capabilities of Large Language Models (LLMs).

The framework consists of three core stages (illustrated in Figure 2):

A. Prompt-Guided Context-Aware Modality Imputation (PCMI)

Instead of zero-padding missing data, LIMSSR treats missingness as a variable to be inferred.

• Tokenization: Available modalities are converted into token sequences with special boundary tokens (e.g., <visual_start>, <visual_end>).
• Missing Tokens: Missing modalities are represented by learnable special missing tokens (e.g., <missing_audio>) repeated for the sequence length.
• Prompting: A structured prompt explicitly describes the task state (e.g., "Given {available} features, {missing} is missing. Infer latent representations..."). This guides the LLM to "imagine" the missing semantics based on the available context and its pre-trained priors, rather than pixel-level reconstruction.

B. LLM-Driven Multidimensional Representation Fusion (LMRF)

To integrate the inferred information without disrupting the LLM's generative flow:

• Fusion Tokens: Special fusion tokens (e.g., <emb_dim_1>, ..., <emb_dim_K>) are appended to the sequence.
• Instruction: The prompt instructs the LLM to aggregate multimodal features into these specific slots.
• Role-Aware Weighting: The model learns weights to combine the outputs of these $K$ fusion tokens into a main representation vector ($z_{main}$), allowing the LLM to compress diverse evaluation criteria (e.g., difficulty, execution, artistry) into distinct latent slots.

C. Mask-Aware Dual-Path Aggregation (MDA)

To mitigate LLM hallucinations caused by severe missingness, the output is processed through two parallel paths:

1. Path 1 (Uncertainty-Calibrated Reasoning): Uses the LLM's high-level semantic reasoning ($z_{main}$). A gating mechanism dynamically adjusts this based on the missingness mask to calibrate confidence.
2. Path 2 (Cross-Modal Pattern Recovery): A statistical path that performs temporal pooling and self-attention on available modalities to recover low-level feature patterns.

• Collaborative Decoding: The final score is a weighted sum of the semantic reasoning path and the pattern recovery path, controlled by a learnable synergy coefficient ($\lambda_{syn}$). This balances high-level inference with data-driven evidence.

D. Optimization Strategies

• Consistency Learning ($L_{con}$): Enforces alignment between the high-level reasoning path and the low-level pattern recovery path.
• Token-level Metric Regularization ($L_{reg}$): Prevents feature collapse among the $K$ fusion tokens by enforcing diversity (maximizing distance between token representations).

3. Key Contributions

1. New Paradigm: The first work to systematically address Training-Time Incomplete Observations in multimodal learning, breaking the dependency on complete training data or reconstruction supervision.
2. LLM as Reasoning Engine: Proposes a novel formulation of IML as a conditional sequence-to-score reasoning task, utilizing LLMs to infer latent semantics for missing modalities via prompt-guided context awareness.
3. Robust Architecture: Introduces the Mask-Aware Dual-Path Aggregation (MDA) mechanism to dynamically balance semantic reasoning and statistical pattern recovery, effectively mitigating hallucinations.
4. State-of-the-Art Performance: Demonstrates that the method outperforms existing SOTA baselines (which rely on full-modal training) even under the more challenging training-time incomplete setting.

4. Experimental Results

The method was evaluated on three long-term AQA benchmarks: FS1000 (Figure Skating), Fis-V (Figure Skating), and Rhythmic Gymnastics (RG).

• Performance: LIMSSR achieved State-of-the-Art (SOTA) results across all datasets and missingness patterns.
  • On FS1000, it improved average Spearman Correlation by 0.9% and reduced MSE by 8.4% compared to the best competing method (MCMoE).
  • On Fis-V, it improved average Sp. Corr. by 1.2% and reduced MSE by 11.3%.
  • Crucially, LIMSSR outperformed methods that had access to complete training data, proving the efficacy of the reasoning-based approach over reconstruction-based approaches.
• Ablation Studies:
  • Removing the LLM backbone caused significant performance drops.
  • Removing PCMI (using zero-padding instead of missing tokens) degraded performance, validating the importance of prompt-guided inference.
  • Removing MDA led to instability, confirming the necessity of dual-path calibration.
• Efficiency: The framework uses a sub-1B parameter LLM (Qwen3-0.6B) with LoRA fine-tuning, making it feasible for edge deployment while maintaining scalability.

5. Significance and Impact

• Real-World Applicability: By removing the requirement for complete training data, LIMSSR aligns with real-world data acquisition constraints where sensors fail or privacy limits data collection.
• Shift in Perspective: The paper shifts the IML paradigm from "reconstructing missing pixels" to "reasoning about missing semantics," leveraging the inherent world knowledge of LLMs.
• Generalizability: While tested on AQA, the framework is designed to be generalizable to other multimodal tasks (e.g., emotion recognition, medical diagnosis) where data is inherently incomplete.
• Data Efficiency: It demonstrates that high-performance multimodal learning is possible with significantly less data supervision, offering a new direction for data-efficient AI.

In conclusion, LIMSSR establishes a new standard for handling incomplete multimodal data by transforming the problem into a reasoning task solvable by LLMs, achieving superior robustness and accuracy without the need for idealized complete training datasets.