CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection

Technical Summary of CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection

Problem Statement
Recent vision-language pre-trained models (VL-PTMs), such as CLIP, have demonstrated remarkable success in open-vocabulary tasks. However, their deployment in real-world scenarios often requires fine-tuning for downstream applications. This process introduces a critical challenge: standard fine-tuning can distort the models' general knowledge, impairing their ability to handle distribution shifts. Real-world environments present two distinct types of distribution shifts:

1. Covariate Shifts (Closed-Set OOD): Changes in image styles or environments where class labels remain the same as the training data (e.g., a dog in a sketch style).
2. Semantic Shifts (Open-Set OOD): The presence of test-time classes that were unseen during training (e.g., a panda when the model was trained on dogs).

Existing fine-tuning methods typically focus on improving robustness to covariate shifts (OOD generalization) or detecting unseen classes (open-set OOD detection) in isolation. They often fail to address both simultaneously. Furthermore, energy-based detection methods often struggle because the energy distributions of closed-set ID data and open-set OOD data significantly overlap, making discrimination difficult. The paper addresses the question: How can VL-PTMs be fine-tuned to improve generalization to closed-set OOD data while simultaneously effectively detecting open-set unseen classes?

Methodology
The authors propose CRoFT, a unified fine-tuning framework that enables the concurrent optimization of OOD generalization and open-set OOD detection. The methodology is built upon three core components:

1. Energy Distribution Reshaping (EDR) Loss:
   Instead of directly minimizing energy scores on in-distribution (ID) data, the authors propose minimizing the gradient magnitude of the energy scores. Theoretically, minimizing the gradient magnitude of energy scores ($\nabla_\theta E_\theta(x) \to 0$) leads to domain-consistent Hessians of the classification loss. This EDR loss is formulated as minimizing the squared norm of the gradient vector of the log-sum-exp of the energy terms. This approach reshapes the energy distribution to better distinguish known classes from unknown ones without sacrificing classification accuracy.

2. Hessian-Based OOD Generalization Bound:
   Through theoretical analysis, the authors establish a link between the EDR loss and OOD generalization. They demonstrate that the EDR loss induces domain-consistent Hessians, which helps bound the generalization performance on closed-set OOD test data under worst-case scenarios. Specifically, they derive a generalization bound showing that the performance gap between source and target domains is related to the empirical risk on worst-case covariate-shifted data and the curvature (Hessian) of the risk function.

3. Concurrent Optimization Framework:
   To minimize the derived generalization bound, CRoFT employs an adversarial learning strategy. It generates "worst-case" covariate-shifted image features that preserve semantic information but differ from the ID features. The framework optimizes a unified objective function comprising:

   • Standard classification loss on ID features.
   • A loss ($L_c$) optimized on the generated worst-case OOD features to enhance robustness.
   • The EDR loss ($L_e$) applied to both ID and generated OOD features to reshape energy distributions.

   The framework utilizes lightweight fine-tuning (injecting one-layer linear projections/adapters into the frozen CLIP encoders) to ensure computational efficiency.

Key Contributions

• Unified Framework: CRoFT is the first framework to simultaneously optimize for closed-set OOD generalization and open-set OOD detection during the fine-tuning of VL-PTMs.
• Theoretical Insight: The paper provides a theoretical proof that minimizing the gradient magnitude of energy scores (EDR loss) leads to domain-consistent Hessians, thereby theoretically bounding the OOD generalization error. This reveals a hidden connection between OOD detection and OOD generalization.
• Novel Loss Function: The introduction of the EDR loss, which reshapes energy distributions by minimizing gradient magnitudes rather than just energy values, improves the separation between ID and OOD data.
• Adversarial Generation: The method generates worst-case covariate-shifted features to explicitly train the model against distribution shifts, improving robustness.

Experimental Results
Extensive experiments were conducted on two setups:

• Setup-I: Large-scale ImageNet variants with 40% closed-set classes and 60% open-set classes, including covariate-shifted datasets (ImageNet-A, R, Sketch, V2).
• Setup-II: Cross-dataset evaluation using domain generalization datasets (PACS, VLCS) as closed-set and other datasets (Caltech101, DTD, Food101) as open-set.

Key Findings:

• Open-Set Detection: CRoFT significantly outperforms state-of-the-art fine-tuning methods (e.g., CoOp, CoCoOp, CLIP-Adapter, Tip-Adapter-F). In Setup-I, it achieved up to a 25.3% reduction in FPR95 (False Positive Rate at 95% recall) compared to zero-shot CLIP, and up to 20% improvement in detecting open-set unseen classes compared to other fine-tuning methods.
• OOD Generalization: CRoFT demonstrates comparable or superior generalization performance on closed-set OOD test sets (e.g., ImageNet-Sketch, ImageNet-A) compared to existing methods.
• Ablation Studies: Removing either the EDR loss ($L_e$) or the adversarial loss ($L_c$) resulted in degraded performance on both tasks, validating the necessity of concurrent optimization.
• Feature Visualization: t-SNE visualizations and energy distribution plots show that CRoFT successfully reduces the overlap between ID and OOD energy distributions and learns more distinct feature representations for open-vs-closed discrimination.

Significance and Claims
The paper claims that CRoFT addresses a critical gap in the deployment of VL-PTMs by enabling them to handle both known distribution shifts and unknown classes simultaneously. The authors emphasize that their theoretical discovery—that optimizing for open-set detection via EDR loss inherently improves OOD generalization—provides a new direction for robust fine-tuning. The method is presented as a practical solution for high-risk applications (e.g., autonomous driving, medical imaging) where detecting unseen classes is as crucial as maintaining accuracy on known classes under environmental changes. The code is made available to facilitate further research in robust vision-language model adaptation.