PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization
The paper proposes PACE, a novel parameter-efficient fine-tuning method that enhances model generalization and preserves pre-trained knowledge by employing consistency regularization with multiplicative noise to implicitly reduce gradient norms and align fine-tuned models with their pre-trained counterparts.