🤖 machine learning

Certification of Machine Learning Models via Directional Sharpness

This paper introduces "directional sharpness," a computationally efficient and reliable metric that outperforms existing measures in certifying machine learning model generalization, even when training processes are perturbed or data privacy must be preserved via zero-knowledge proofs.

Original authors: Gefei Tan, Adria Gascon, Sarah Meiklejohn, Mariana Raykova

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gefei Tan, Adria Gascon, Sarah Meiklejohn, Mariana Raykova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a chef to cook a massive banquet. You want to be sure the food will taste good not just for the few people you taste-tested it on, but for everyone who eats it later. In the world of machine learning, this is called generalization: the ability of a computer model to perform well on new, unseen data.

The problem is, how do you check if the chef is truly talented, or if they just memorized the specific dishes you gave them for the taste test?

This paper introduces a new way to check a model's "cooking skills" called Directional Sharpness. Here is the breakdown using simple analogies.

The Problem: The "Flat" Trap

In machine learning, we often look at a model's "loss landscape." Imagine this as a hilly terrain where the bottom of a valley represents a perfect model.

The Goal: We want the model to sit in a wide, flat valley. If the valley is wide, the model is stable. Even if you nudge it slightly (like a new piece of data), it stays in the valley and keeps working well.
The Trap: Sometimes, a model sits in a narrow, sharp spike that looks like a valley from a distance. It looks perfect on the data it was trained on, but if you nudge it even a tiny bit, it falls off the spike and crashes. This is "overfitting" or a "backdoor" (a hidden flaw).

The Old Way: Taking a Single Snapshot

Previously, experts tried to measure how "flat" a valley was by taking a single, static photo. They would look at the model, give it one tiny nudge, and see how much the error went up.

The Flaw: This is like looking at a mountain peak from one angle. You might miss the fact that the peak is actually a thin needle that will collapse if you look at it from a different angle. If a model is "cheating" (e.g., it has a hidden backdoor or memorized the data), a single snapshot might still look flat, fooling the inspector.

The New Solution: Directional Sharpness

The authors propose Directional Sharpness. Instead of taking one photo, imagine you are rolling a ball through the valley while shaking the ground slightly.

The Dynamic Test: You don't just look at the model once. You apply a series of small, random nudges (like a gentle earthquake) and watch how the model reacts over time.
The Stable Model: If the model is truly good (in a wide, flat valley), it will wobble a little but stay calm. The "sharpness" score remains low and steady.
The Cheating Model: If the model is sitting on a narrow spike or has hidden flaws, those small nudges will eventually push it off balance. The "sharpness" score will start to spike and fluctuate wildly.

The Analogy:

Old Method: Checking if a tightrope walker is balanced by asking them to stand still for one second.
New Method (Directional Sharpness): Asking the tightrope walker to walk the rope while a wind machine blows randomly at them. If they wobble and fall, you know they aren't ready, even if they stood still perfectly for that one second.

Why This Matters

The paper claims this new method is better for three reasons:

It's a Better Predictor: It correlates much more strongly with whether a model will actually work on new data than the old "snapshot" methods. It catches models that look good but are actually fragile.
It Catches Cheaters: It is very good at spotting models that have been "poisoned" (trained with bad data) or have "backdoors" (hidden triggers). These models often look perfect on a standard test but fail the "shaking" test.
It's Fast and Private:
- Speed: It is much faster to calculate than the old methods (up to 4 times faster than just checking test accuracy in some cases).
- Privacy: You can use a cryptographic trick called a "Zero-Knowledge Proof" to prove the model passed this test without the verifier ever seeing the secret training data. It's like proving you have a winning lottery ticket without showing the ticket to anyone.

The Bottom Line

The paper argues that to truly trust an AI, we shouldn't just ask, "Did you get the right answer on the practice test?" We should ask, "Can you stay balanced when the ground starts shaking?"

Directional Sharpness is the tool that shakes the ground and tells us if the model is truly stable or just pretending to be. This helps auditors and companies ensure that AI models are safe, reliable, and not hiding any nasty surprises.

Technical Summary: Certification of Machine Learning Models via Directional Sharpness

Problem Statement

Model certification aims to provide assurance regarding the trustworthiness and quality of machine learning models, specifically their ability to generalize to unseen data. Directly measuring generalization is impossible because it depends on unknown data distributions. Existing proxies, such as test accuracy, are unreliable as they can be misleading due to dataset errors, overfitting, or distribution shifts. Furthermore, theoretical generalization bounds are often too loose or vacuous for modern over-parameterized networks.

While "sharpness" (the sensitivity of loss to parameter perturbations) has emerged as a promising empirical proxy for generalization, existing sharpness metrics face two critical barriers in certification contexts:

Reliability under Deviation: Current metrics are validated primarily on honestly trained models. It is unclear if they can detect poor generalization when training deviates from prescribed procedures (e.g., due to backdoors, overfitting, or label noise).
Computational Efficiency: Existing metrics are computationally expensive, making them impractical for model auditing (where a verifier has access to training data) and infeasible for model certification (where a verifier must check a cryptographic proof without access to training data).

Methodology: Directional Sharpness

The authors propose Directional Sharpness, a metric designed to measure the dynamic stability of a model rather than its static geometric properties.

Core Concept

Unlike static sharpness, which evaluates the loss landscape at a single point (often averaging over the full dataset), directional sharpness assesses how a model behaves under a sequence of continuous, stochastic perturbations. The intuition is that a model in a genuinely flat, wide basin will remain stable under these perturbations, whereas a model in a narrow, unstable pocket (or one with incoherent geometry) will be pushed into sharp regions, exhibiting detectable fluctuations.

Algorithm

The computation of Directional Sharpness (Algorithm 1) involves:

Initialization: Start with a trained model $w_0$ .
Probing Trajectory: Apply $T$ $T$ steps of Sharpness-Aware Minimization (SAM)-style updates. At each step $t$ $t$ :
- Sample a mini-batch $\xi_t$ from the training data.
- Compute the per-step sharpness $s_t$ (e.g., ASAM sharpness) on the mini-batch.
- Update the model parameters $w_{t+1}$ using the SAM update operator.
Fluctuation Statistic: Record the sequence of sharpness values $\{s_0, \dots, s_{T-1}\}$ . The final metric is a fluctuation statistic (specifically, the standard deviation of the log-ratio of squared sharpness to loss) of this sequence.

Theoretical Grounding

The paper provides theoretical analysis linking directional sharpness to SAM-stability:

Stability Bound: If a model resides in a SAM-stable minimum, the expected squared sharpness under continuous perturbation is uniformly bounded by a constant multiple of the initial training loss.
Instability Detection: Conversely, if the expected squared sharpness grows exponentially, the model resides in an unstable region.
Gradient Coherence: The authors formalize a class of training deviations characterized by low gradient coherence (where per-example gradients cancel out in the full-batch average but remain large individually). They prove that while full-batch static sharpness may appear low (masking the deviation), the root-mean-square (RMS) mini-batch sharpness remains high. Directional sharpness, by aggregating mini-batch signals over time, amplifies this gap, making it sensitive to such deviations.

Key Contributions

New Metric (Directional Sharpness): A generalization metric that shifts from static geometric measurement to dynamic stability measurement.
Superior Correlation: Empirical evidence showing directional sharpness correlates significantly more strongly with the generalization gap than existing static metrics (e.g., Worst-Case Sharpness, ASAM).
Robustness to Deviations: Demonstrated ability to identify models with poor generalization caused by training deviations (label noise, overfitting to spurious features, backdoors) where test accuracy and static sharpness fail.
Efficiency for Certification:
- Auditing: Computation is up to $4\times$ faster than computing test accuracy and orders of magnitude faster than static sharpness.
- Certification (Zero-Knowledge): The authors demonstrate how to generate Zero-Knowledge Proofs (ZKPs) for directional sharpness. They show that proving directional sharpness is up to 80,000 $\times$ faster than proving the entire training process (zkPoT) for models like VGG-11.

Experimental Results

Correlation with Generalization

In a large-scale experiment training 1,152 models on CIFAR-10/100 with varying architectures and hyperparameters:

Directional sharpness achieved a Spearman correlation ( $\rho$ ) of 0.902 with the generalization gap.
This significantly outperformed the best static baselines, such as ASAM with full-batch ( $\rho \approx 0.64$ ) and Worst-Case Sharpness ( $\rho \approx 0.60$ ).

Reliability Under Training Deviations

The authors conducted distinguishability experiments pairing benign models with faulty models that had nearly identical observed test accuracy. Directional sharpness successfully separated the models in all scenarios, whereas static metrics and test accuracy failed:

Noisy Labels: Benign/Faulty ratio of 0.177 (Directional) vs. 0.991 (Static).
Overfitting to Spurious Features: Ratio of 0.084 (Directional) vs. 0.963 (Static).
Backdoors: Ratio of 0.061 (Directional) vs. 0.587 (Static).
Quantization and Small Test Sets: Directional sharpness consistently provided stronger separation margins.

Efficiency and ZKP

Runtime: Computing directional sharpness on a small batch took 0.21 seconds, compared to 0.92 seconds for test accuracy and 210.32 seconds for static Worst-Case Sharpness.
ZKP Cost: For a VGG-11 model trained over 150 epochs, a ZKP of directional sharpness took 88.87 minutes, whereas a ZKP of the full training process (LATE-SAM baseline) was estimated to take 118,671 hours (approx. 13.5 years).

Significance and Claims

The paper claims that directional sharpness provides an efficient and robust metric for evaluating model generalization. Its significance lies in enabling practical model certification in two settings:

Auditing: Where a trusted verifier with access to training data can efficiently detect poor generalization and training deviations.
Certification: Where an external verifier can cryptographically verify model quality via ZKPs without accessing the training data, overcoming the prohibitive computational costs of existing training proofs.

The authors assert that by measuring dynamic stability, directional sharpness offers a more reliable signal of model quality than static geometric properties, particularly in adversarial or non-ideal training scenarios. They conclude that this metric is a suitable ingredient for mitigating attacks in adversarial machine learning and improving transparency in ML systems.