Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

1. Problem Definition & Motivation

The paper addresses Zero-Shot Compositional Action Recognition (ZS-CAR), a task where a model must recognize novel verb-object combinations (e.g., "open drawer") using a shared vocabulary of verbs and objects, even if that specific pair was never seen during training.

The Core Failure Mode:
The authors identify a critical failure mode where models rely on object-driven shortcuts rather than temporal evidence. Instead of analyzing the motion to determine the verb (e.g., distinguishing "opening" from "closing"), the model predicts the verb based on the labeled object class and the statistical co-occurrence of that object with specific verbs in the training data.

• Example: If a model sees a "drawer" in the training set mostly paired with "opening," it will predict "opening" for an unseen "close drawer" video simply because it recognizes the drawer, ignoring the temporal motion.

Root Causes Identified:

1. Sparse and Skewed Compositional Supervision: Datasets contain only a tiny fraction of possible verb-object combinations. Training data is heavily skewed toward frequent pairs, creating strong co-occurrence priors.
2. Asymmetric Learning Difficulty: Objects are visually explicit and can be recognized from a single frame (static cues). Verbs require multi-frame temporal reasoning. Models naturally gravitate toward the "easier" cue (the object) to minimize loss, leading to shortcut learning.

2. Methodology: RCORE

The authors propose RCORE (Robust COmpositional REpresentations), a diagnosis-driven framework designed to mitigate these shortcuts through two complementary regularization components.

A. Co-occurrence Prior Regularization (CPR)

CPR aims to break the model's reliance on frequent training pairs by expanding supervision to unseen compositions.

• Synthesized Composition Supervision: The method synthesizes new training samples by injecting static object cues from one video into the high-motion regions of another. This creates plausible but unseen verb-object pairs (e.g., taking a "lifting" action video and swapping the object to a new one).
• Hard Negative Regularization: The model is explicitly penalized for over-scoring frequent, seen compositions that act as "hard negatives" for the synthesized unseen pairs. This forces the model to distinguish between the synthesized target and the dominant co-occurrence priors.
• Batch-Adaptive Label Space: Instead of optimizing over the entire (and mostly empty) verb-object space, CPR dynamically expands the label space for each mini-batch to include synthesized pairs, ensuring stable optimization while injecting supervision for unseen compositions.

B. Temporal Order Regularization for Composition (TORC)

TORC enforces the model to learn verb representations grounded in temporal dynamics rather than static object appearance.

• Temporal Perturbation: The model is fed two perturbed views of the input: a reversed sequence and a shuffled sequence.
• Cosine Similarity Loss ($L_{cos}$): The model is penalized if the feature representation of the original video is too similar to the reversed video. Since reversing time often flips the meaning of an action (e.g., opening vs. closing), high similarity indicates a failure to capture temporal semantics.
• Entropy Loss ($L_{ent}$): When the temporal order is disrupted (shuffled), the model is encouraged to produce high-entropy (uncertain) verb predictions. This prevents the model from confidently predicting a verb based solely on static spatial cues.

3. Diagnostic Metrics

To quantify the problem and the solution, the authors introduce specific diagnostic metrics:

• False Seen Prediction (FSP): The fraction of unseen inputs incorrectly predicted as a seen composition.
• False Co-occurrence Prediction (FCP): The fraction of unseen inputs predicted as a frequent seen composition.
• Compositional Gap ($\Delta_{CG}$): Measures the benefit of joint modeling over independent recognition. A negative gap indicates the model performs worse when predicting the composition jointly than when predicting the verb and object separately, signaling a failure in compositional reasoning.

4. Key Contributions

1. Diagnosis of Object-Driven Shortcuts: The paper empirically proves that ZS-CAR models fail due to asymmetric learning difficulty and skewed co-occurrence priors, leading to a reliance on object cues for verb prediction.
2. RCORE Framework: A novel training framework combining CPR (to suppress co-occurrence bias via synthesized supervision) and TORC (to enforce temporal sensitivity).
3. New Benchmark (EK100-com): The authors curate EK100-com from EPIC-KITCHENS-100, a dataset with even more severe compositional sparsity (7.5% label coverage) than the standard Sth-com, providing a more rigorous testbed.
4. Open-World Evaluation Protocol: The authors advocate for an open-world evaluation protocol (inference over the full verb-object space without masking unseen pairs) to expose true generalization capabilities, contrasting with the closed-world protocols used in prior work.

5. Experimental Results

The method was evaluated on Sth-com (Something-Something) and the new EK100-com datasets using CLIP and InternVideo2 backbones.

• Performance Gains: RCORE significantly outperforms the state-of-the-art baseline (C2C).
  • On Sth-com, it improves unseen composition accuracy by +3.8% (CLIP) and +4.5% (InternVideo2).
  • On EK100-com, it achieves a massive +6.9% (CLIP) and +7.7% (InternVideo2) improvement in unseen accuracy.
• Reduction in Shortcuts:
  • RCORE drastically reduces FSP and FCP rates during training, indicating less reliance on seen co-occurrence patterns.
  • It achieves a positive Compositional Gap ($\Delta_{CG}$) on unseen compositions, whereas baselines remain negative, proving genuine compositional reasoning.
• Temporal Sensitivity:
  • When tested on temporally shuffled inputs, RCORE shows a significantly larger performance drop than the baseline, confirming it relies on temporal dynamics rather than static cues.
  • Cosine similarity analysis shows RCORE successfully distinguishes between reversed actions (e.g., opening vs. closing), while baselines fail to do so.

6. Significance

This work fundamentally shifts the understanding of failure modes in video action recognition. It demonstrates that simply using stronger video backbones (like InternVideo2) is insufficient to solve the compositional generalization problem if the training objective allows for object-driven shortcuts.

By explicitly regularizing against co-occurrence priors and enforcing temporal order sensitivity, RCORE provides a robust solution for Zero-Shot Compositional Action Recognition. The findings suggest that for models to generalize to real-world scenarios (where unseen combinations are common), they must be trained to ignore static object biases and focus on the temporal dynamics of human actions. The introduction of the EK100-com dataset and the open-world evaluation protocol sets a new standard for future research in this field.