💻 computer science

TraMP-LLaMA: Generative Interpretability with Decoupled Instruction Tuning for Facial Expression Quality Assessment

TraMP-LLaMA is a novel multimodal framework that enhances interpretability in Parkinson's disease assessment by jointly predicting facial expression severity scores and generating structured textual reports through a decoupled instruction-tuning strategy on the newly constructed PFED5-plus dataset.

Original authors: Shuchao Duan, Alan Whone, Hossein Rahmani, Jun Liu, Majid Mirmehdi

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuchao Duan, Alan Whone, Hossein Rahmani, Jun Liu, Majid Mirmehdi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Black Box" Score

Imagine a doctor watching a video of a patient trying to smile or squeeze their eyes shut to check for Parkinson's disease. Currently, the computer software they use acts like a strict grading machine. It watches the video and spits out a single number, like a "Severity Score of 2 out of 4."

While that number is useful for tracking progress over time, it's frustratingly vague. It's like a teacher giving a student a "C" on an essay without writing any comments. You know the grade, but you don't know why. Was the "C" because the handwriting was messy? Because the grammar was wrong? Or because the student didn't show up?

In medical terms, two patients might get the same "Level 2" score, but for totally different reasons. One might have stiff eyelids, while the other has a drooping mouth. The current software can't explain the difference. This makes it hard for doctors to trust the machine or review its work later.

The Solution: The "Bilingual Translator"

The authors created a new AI system called TraMP-LLaMA. Think of this system not just as a grader, but as a bilingual translator who speaks two languages fluently:

The Language of Numbers: It still calculates the severity score.
The Language of Stories: It writes a short, structured report explaining exactly what it saw in the video to justify that score.

Instead of just saying "Level 2," it says: "Level 2. The patient's eyelids tightened strongly, but their mouth remained mostly still." This turns a mysterious number into a transparent, auditable story.

How It Works: The "Two-Headed Brain"

To make this work, the AI needed to learn two difficult things at once without getting confused. The authors designed a special "brain" with two distinct heads:

The Motion Detective (The Score Head): This part looks at the video and tracks the tiny movements of facial landmarks (like the corners of the eyes and mouth). It focuses purely on the math to get the score right.
The Storyteller (The Report Head): This part is a large language model (like a very smart chatbot). It takes the evidence found by the detective and writes a human-readable report.

The Secret Sauce: The "Stop-Gradient" Switch
Usually, if you try to teach a student to do math and write poetry at the same time, they might get mixed up. The math might ruin the poetry, or the poetry might distract from the math.

The authors solved this with a clever trick they call Decoupled Instruction Tuning. Imagine a one-way glass between the two heads:

The "Storyteller" can look at the "Detective's" notes to write its report.
BUT, the "Detective" is protected. If the "Storyteller" makes a mistake in its writing, that error cannot flow back and mess up the "Detective's" math calculations.

This ensures the severity score remains perfectly accurate (like a strict math teacher) while still allowing the system to generate a helpful explanation (like a creative writer).

The New Dataset: PFED5+

To teach this AI how to write these reports, the authors had to create a new textbook. They took an existing dataset of facial videos (PFED5) and added expert-written descriptions to every single clip.

Think of this as hiring a team of clinical editors. They watched every video and wrote down exactly what happened, phase by phase:

Before the action: "The face was neutral."
During the action: "The eyebrows stayed still, but the cheeks raised with wrinkles."
After the action: "The face relaxed back to neutral."

They made sure these descriptions were purely factual observations (what you can see) rather than guesses about feelings. This created a new dataset called PFED5+, which the AI used to learn how to match video evidence with the right words.

The Results: Better Scores and Better Stories

When they tested TraMP-LLaMA against other top AI models:

For Scoring: It became the best at predicting the severity score, beating previous methods by a significant margin (improving accuracy by at least 4.39%).
For Reporting: It wrote better, more accurate reports than standard video-chatbots, which often hallucinate (make things up) or miss the subtle details needed for medical checks.

The Bottom Line

TraMP-LLaMA is a step forward in making AI medical tools trustworthy. It doesn't just give a grade; it shows its work. By separating the math from the storytelling, it ensures the diagnosis is accurate while finally giving doctors the "why" behind the "what."

Note: The paper focuses on the technical creation of this tool and its performance on a specific dataset. It does not claim the tool is currently being used in hospitals or that it replaces human doctors, but rather that it provides a new way to make facial analysis more transparent.

Technical Summary: TraMP-LLaMA: Generative Interpretability with Decoupled Instruction Tuning for Facial Expression Quality Assessment

Problem Statement
Existing Facial Expression Quality Assessment (FEQA) methods for neurological disorders, such as Parkinson's disease (PD), typically output a single severity score (e.g., MDS-UPDRS). While useful for quantitative monitoring, these scores lack explicit communication of the observable facial motion evidence supporting the prediction. This limits interpretability and hinders the ability to audit model outputs or analyze rater disagreements. Traditional interpretability tools like saliency maps fail to describe clinically relevant spatio-temporal motor patterns (e.g., reduced range of motion, asymmetry), while off-the-shelf Video-Language Models (VLMs) often hallucinate clinically irrelevant details when asked to justify predictions. Furthermore, existing datasets like PFED5 provide severity scores but lack the textual annotations of the underlying motor evidence required to train models for evidence-grounded reporting.

Methodology
The authors propose TraMP-LLaMA, a unified multimodal framework designed to jointly predict clinical severity scores and generate structured, evidence-based textual reports.

Architecture: The framework builds upon VideoLLaMA3 (using SigLIP-NaViT as the vision encoder and Qwen-2.5 as the LLM). It integrates two primary input streams:
1. RGB Appearance: Encoded by a frozen SigLIP-NaViT vision encoder.
2. Landmark Trajectories: Encoded by a trainable SkateFormer network to capture explicit facial motion features.
  These streams are processed through a trainable cross-fusion module to create a compact, motion-aware representation.
Dual-Head Output:
1. Severity Scoring: A regression head predicts the clinical score based on the fused trajectory-RGB representation.
2. Report Generation: The LLM is instruction-tuned to generate structured reports. It conditions on the concatenation of visual tokens, trajectory-based motion tokens, and the fused representation, alongside a specific instruction prompt.
Decoupled Instruction Tuning: A core technical contribution is the mitigation of task interference between severity regression and language generation. The authors employ a stop-gradient mechanism where:
- The motion encoder and cross-fusion module are updated only by the severity scoring loss ( $L_{score}$ ).
- Gradients from the text generation loss ( $L_{text}$ ) are blocked from flowing back into the motion encoder and fusion module.
- The LLM parameters (adapted via LoRA) and projection layers are updated only by $L_{text}$ .
  This design ensures that the spatio-temporal features required for accurate scoring are not distorted by the language generation objective, while still allowing the LLM to utilize the same evidence for report generation.
Dataset Extension (PFED5+): To support report generation, the authors extended the PFED5 dataset to PFED5+. This involved creating expert-guided, clinically reviewed textual descriptions for each video clip. These annotations follow a standardized temporal template (Pre-action, In-action, Post-action) and describe observable, region-specific motor patterns without directly encoding the severity label, thereby preventing label leakage.

Key Contributions

TraMP-LLaMA Framework: A multimodal system that simultaneously outputs quantitative severity scores and structured textual reports, providing inspectable facial motion evidence alongside assessments.
Decoupled Training Strategy: A novel instruction-tuning approach that blocks language-generation gradients from updating the motion encoder and fusion module, preserving scoring accuracy while enabling generative interpretability.
PFED5+ Dataset: An annotation-augmented version of PFED5 containing expert-guided, clip-level motion descriptions, enabling the first supervised training of evidence-grounded report generation for FEQA.

Results
Experiments were conducted on PFED5+ using a joint multi-expression training setting (training a single model across all five clinical actions).

Severity Prediction: TraMP-LLaMA achieved state-of-the-art performance, outperforming competitive baselines including QAFE-Net, TraMP-Former, and the base VideoLLaMA3. Specifically, it improved Spearman's rank correlation by at least 4.39% over all competing methods in the joint setting. The inclusion of trajectory cues via cross-fusion provided significant gains, particularly for actions like 'Sit at rest' (+10.40%) and 'Squeeze eyes' (+9.76%).
Report Generation: TraMP-LLaMA surpassed the base VideoLLaMA3 and other general-purpose VLMs (e.g., Chat-UniVi, LLaVA-Video) across all metrics (BERTScore, ROUGE-L, BLEU-4, CIDEr). The ablation studies confirmed that the decoupled training strategy (blocking text gradients) was critical, as enabling these gradients resulted in significantly lower scoring correlation and unstable training.
Ablation Findings: The study validated that using a combination of visual, trajectory, and fused tokens as LLM conditioning yields the best semantic alignment. Global Average Pooling (GAP) for temporal aggregation was found to be more effective than frame-wise sequences for overall scoring correlation.

Significance and Limitations
The paper claims that TraMP-LLaMA addresses a critical gap in clinical FEQA by moving beyond "black box" scoring to provide transparent, auditable, and evidence-grounded assessments. By linking ratings to specific visible motor evidence, the framework supports longitudinal comparison and rater analysis.

However, the authors maintain a modest stance regarding current limitations:

As a generative VLM, the model may occasionally hallucinate region-specific details, particularly when motion cues are subtle or occluded.
The PFED5+ dataset remains relatively small and imbalanced across severity levels, with report supervision constrained by single-reference annotations and inter-rater variability.
While metrics improve, the authors note that format deviations and unsupported region statements can still occur, which may limit immediate clinical usability despite improved overlap-based metrics.