🤖 machine learning

PVeRA: Probabilistic Vector-Based Random Matrix Adaptation

This paper introduces PVeRA, a probabilistic extension of the VeRA adapter that modifies its shared low-rank matrices to better handle input ambiguities and achieve superior performance on the VTAB-1k benchmark compared to existing parameter-efficient adaptation methods.

Original authors: Leo Fillioux, Enzo Ferrante, Paul-Henry Cournède, Maria Vakalopoulou, Stergios Christodoulidis

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leo Fillioux, Enzo Ferrante, Paul-Henry Cournède, Maria Vakalopoulou, Stergios Christodoulidis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly smart library of knowledge (a "foundation model") that has read almost every book in the world. This library is brilliant, but it's also huge, heavy, and expensive to move around. If you want to teach it a specific new skill—like recognizing rare flowers or diagnosing a specific medical condition—you usually have to "fine-tune" it.

The Problem: Traditional fine-tuning is like trying to rewrite the entire encyclopedia to fit your new topic. It takes a lot of time, money, and computing power. Plus, if you only have a few examples of the new topic, the library might get confused and forget its old knowledge (overfitting).

The Current Solution (Adapters): To fix this, scientists invented "adapters." Think of these as small, lightweight sticky notes you attach to the library's shelves. Instead of rewriting the books, you just add these notes to guide the library on how to handle your new task. One popular type of sticky note is called VeRA. It uses a pair of "frozen" (unchanging) random patterns shared across the whole library, and it only learns a few tiny numbers (vectors) to adjust how those patterns are used. It's efficient, but it's a bit rigid—it makes one single, fixed decision for every input.

The New Idea: PVeRA
The authors of this paper propose PVeRA (Probabilistic Vector-Based Random Matrix Adaptation). Here is the simple breakdown of what they did:

1. From a Single Guess to a "Cloud of Possibilities"

Imagine you are trying to identify an animal in a foggy picture.

Old Way (VeRA): The model looks at the fog and says, "I am 100% sure this is an elephant." It makes one rigid guess based on a fixed rule.
New Way (PVeRA): The model looks at the fog and says, "This looks mostly like an elephant, but there's a small chance it could be a horse, and the fog makes me a little unsure."

PVeRA changes the "sticky notes" so they don't just hold one number; they hold a distribution (a cloud of possibilities). Instead of picking a single fixed adjustment, the model learns a range of possible adjustments. During training, it randomly samples from this cloud, effectively practicing with many slightly different versions of itself. This helps it handle the "fog" (ambiguity) in the data much better.

2. Why This is Like a "Safety Net"

The paper claims this probabilistic approach gives the model two superpowers:

Confidence Intervals (The "How Sure Are You?" Meter):
Because the model samples from a cloud of possibilities, you can ask it to look at the same image 10 times.
- If it says "Elephant" 10 times with the same confidence, you know it's certain.
- If it says "Elephant" 6 times, "Horse" 3 times, and "Zebra" 1 time, you know it's unsure.
  The paper shows that PVeRA can naturally generate these "confidence intervals" without needing extra, complicated math. It's like the model has a built-in honesty meter that tells you when it's guessing.
Better Performance with Less Data:
The authors tested this on 19 different datasets (ranging from natural photos to medical images and structured data). They found that PVeRA consistently performed better than the original VeRA and other popular adapters. It was especially good at being stable across different types of tasks.

3. The "Magic" of Merging

One of the coolest features mentioned is that even though PVeRA uses a "cloud" of possibilities during training, you can still merge it into the main model for real-world use.

Analogy: Imagine you practiced a speech by trying out 100 different versions of your delivery. Once you've practiced enough, you pick the best version of your speech and memorize it.
Result: When you actually give the speech (inference), you don't need to try 100 versions. You just deliver the one polished version. This means PVeRA is just as fast as the old methods when you actually use it, but it learned smarter during training.

4. Spotting the "Outsiders"

The paper also found that because PVeRA learns a distribution, it's better at spotting things it doesn't know.

Analogy: If you show a model trained on cats and dogs a picture of a toaster, a rigid model might confidently say "Cat" (because it looks a bit like a cat). A PVeRA model, however, might say, "I'm really unsure about this; my internal confidence is all over the place."
The authors showed that the model's internal "uncertainty signals" were much stronger for things it hadn't seen before (out-of-distribution), making it a better tool for safety-critical tasks where you need to know when the model is confused.

Summary

The paper introduces PVeRA, a smarter, more flexible version of an existing tool called VeRA. By teaching the model to think in terms of "probabilities" and "ranges of possibilities" rather than single fixed answers, it:

Performs better on a wide variety of tasks.
Knows when it doesn't know (better uncertainty estimation).
Stays fast because the complex math is only used during training, not when the model is actually working.

It's essentially upgrading the model's "sticky notes" from rigid instructions to flexible, probabilistic guides that help the model navigate ambiguity with confidence.

1. Problem Statement

Large foundation models (e.g., Vision Transformers) have achieved state-of-the-art performance but require massive computational resources and datasets for fine-tuning. Traditional full fine-tuning is often infeasible due to cost and the risk of overfitting in low-data regimes.

Parameter-Efficient Fine-Tuning (PEFT): Methods like LoRA and VeRA address this by freezing the backbone and training only a small number of parameters (adapters).
Limitations of Current Methods: While VeRA (Vector-based Random matrix Adaptation) is highly efficient by using frozen random low-rank matrices shared across layers, it remains a deterministic approach. It produces a single point estimate for weight adaptations, which limits its ability to:
1. Model inherent ambiguities in input data.
2. Quantify prediction uncertainty.
3. Provide confidence intervals for outputs.

2. Methodology: PVeRA

The authors propose PVeRA, a probabilistic extension of VeRA that treats the low-rank adaptation matrices as random variables drawn from a learned distribution rather than fixed values.

Core Mechanism

Probabilistic Formulation: Instead of learning fixed scaling vectors, PVeRA learns the parameters of a multivariate normal distribution ( $\mu$ and $\sigma$ ) for the latent adaptation space.
Architecture:
- It utilizes frozen random matrices $A$ and $B$ (shared across layers) similar to VeRA.
- It introduces trainable vectors $d$ and $b$ to generate the mean ( $\mu$ ) and standard deviation ( $\sigma$ ) of the distribution.
- Reparameterization Trick: During training, the latent variable $z$ is sampled as $z = \mu + \epsilon \cdot \sigma$ , where $\epsilon \sim \mathcal{N}(0, 1)$ . This allows gradients to flow through the sampling process.
Loss Function: The total loss combines the task-specific classification loss with a Kullback-Leibler (KL) divergence term. The KL term enforces a standard normal prior ( $\mathcal{N}(0, I)$ ) on the learned distributions to prevent overfitting and ensure the latent space remains regularized.
$\mathcal{L}_{total} = \mathcal{L}_{classification} + \beta \sum \mathcal{L}_{KL}$

Inference Strategies

PVeRA supports two inference modes:

Deterministic: Uses only the mean ( $\mu$ ) for the adaptation. The weights can be merged into the backbone, resulting in zero inference overhead (identical to VeRA/LoRA).
Probabilistic (Monte Carlo): Samples multiple times from the learned distribution during inference. This allows for:
- Uncertainty Quantification: High variance in predictions indicates uncertainty.
- Confidence Intervals: Generating statistical confidence intervals for predictions.
- Out-of-Distribution (OOD) Detection: OOD samples typically exhibit different distributional characteristics in the latent space compared to in-distribution samples.

3. Key Contributions

Proposal of PVeRA: A novel adapter that learns a distribution over weight adaptations, introducing an inductive bias to handle feature space ambiguities.
Performance Superiority: Demonstrated that PVeRA outperforms the original VeRA and other state-of-the-art adapters (LoRA, DoRA, IA3, Bottleneck) on average.
Uncertainty & Calibration:
- PVeRA maintains excellent model calibration (predicting accurate confidence scores).
- It enables the generation of confidence intervals and uncertainty estimation without retraining the backbone.
Efficiency: Retains the extreme parameter efficiency of VeRA (orders of magnitude fewer parameters than standard adapters) and allows for weight merging to eliminate inference latency.
Open Source: Provided code for training and benchmarking PVeRA and other adapters.

4. Experimental Results

The authors evaluated PVeRA on the VTAB-1k benchmark (19 datasets across natural, specialized, and structured categories) using a frozen DINOv2 (ViT-B/14) backbone.

Accuracy: PVeRA achieved the highest average accuracy (71.4%) across all 19 datasets, outperforming VeRA (69.9%), LoRA (70.5%), and DoRA (71.0%). It showed particularly stable performance across diverse domains.
Parameter Efficiency: PVeRA maintains a very low parameter count (comparable to VeRA), significantly lower than Bottleneck, AdaptFormer, and DoRA.
Calibration: Using the Adaptive Calibration Error (ACE) metric, PVeRA (along with LoRA and VeRA) showed superior calibration compared to other adapters, meaning its confidence scores better reflect actual accuracy.
Uncertainty Estimation:
- Experiments showed that incorrectly classified samples had significantly higher standard deviations in their softmax scores compared to correctly classified ones when using probabilistic inference.
- Confidence Intervals: Correctly classified samples had much narrower confidence intervals (0.085) compared to incorrect ones (0.225).
Out-of-Distribution (OOD) Detection: The sum of the means ( $\mu_q + \mu_v$ ) in the adapter layers served as an effective metric to distinguish in-distribution from out-of-distribution data, a capability VeRA lacks.
Ablation Studies:
- Rank: A rank of $r=256$ provided the best balance of performance and parameters.
- Position: Applying PVeRA to the Query (Q) and Value (V) branches yielded the best results, consistent with LoRA's original findings.

5. Significance and Impact

PVeRA represents a significant step forward in Parameter-Efficient Fine-Tuning by bridging the gap between efficiency and probabilistic modeling.

Robustness: By modeling uncertainty, PVeRA is better suited for safety-critical applications (e.g., medical imaging) where knowing when a model is unsure is as important as the prediction itself.
Versatility: It extends the utility of VeRA beyond simple classification, enabling tasks like OOD detection and confidence interval estimation without increasing the model's inference cost (when using deterministic merging).
Generalizability: The paper also briefly demonstrated PVeRA's effectiveness on Natural Language Processing tasks (GLUE benchmark), suggesting the probabilistic adaptation mechanism is model-agnostic and applicable beyond computer vision.

In summary, PVeRA offers a "best of both worlds" solution: the computational efficiency and low parameter count of VeRA, combined with the robustness, uncertainty quantification, and improved calibration of probabilistic deep learning.