🤖 AI

APEX: Assumption-free Projection-based Embedding eXamination Metric for Image Quality Assessment

APEX is a novel, assumption-free image quality assessment framework that leverages the Sliced Wasserstein Distance with open-vocabulary foundation models (CLIP and DINOv2) to overcome the limitations of traditional feature-distribution metrics, offering superior robustness and stability across diverse datasets.

Original authors: Caterina Gallegati, Monica Bianchini, Franco Scarselli, Vittorio Murino, Barbara Toniella Corradini

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Caterina Gallegati, Monica Bianchini, Franco Scarselli, Vittorio Murino, Barbara Toniella Corradini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge in a cooking competition. The chefs (generative AI models) are creating incredible new dishes (images) that look almost indistinguishable from real food. Your job is to taste them and decide: "Is this dish good? Is it better than the last one?"

For a long time, the judges used a very old, rigid checklist to grade these dishes. They would look at the ingredients (pixels) and check if they matched a specific recipe from a 1990s cookbook (Inception-v3 trained on ImageNet).

The Problem:
The paper argues that this old checklist has two big flaws:

The "Closed Menu" Problem: The old checklist only knows about ingredients found in its specific 1990s cookbook. If a chef makes a futuristic dish with ingredients the old book never saw, the checklist gets confused or gives a bad score, even if the dish is delicious.
The "Rigid Math" Problem: The old checklist assumes all dishes follow a perfect, predictable bell-curve shape. But real cooking (and real AI images) is messy and complex. Forcing a messy reality into a rigid shape leads to wrong scores.

Recently, some judges tried using a newer, fancier cookbook (Foundation Models like CLIP and DINOv2) to understand modern ingredients. But they kept using the same rigid math to grade them. It's like having a modern menu but still using a 1990s ruler that bends the wrong way.

The Solution: APEX

The authors introduce APEX (Assumption-free Projection-based Embedding eXamination). Think of APEX as a new, super-smart judging system with two main superpowers:

1. The "Shadow Puppet" Trick (Projection-based)
Instead of trying to measure the whole 3D shape of a dish at once (which is hard and requires rigid assumptions), APEX uses a clever trick. Imagine shining a light on a complex sculpture to cast its shadow on a wall.

APEX shines light from thousands of different angles (projections).
It measures the shadow (the 1D slice) each time.
It averages all those shadows to get a true sense of the shape.
Why this matters: This method doesn't assume the sculpture is a perfect sphere or cube. It just measures the shape as it actually is, no matter how weird or complex. It's "assumption-free."

2. The "Universal Translator" (Foundation Models)
Instead of using the old 1990s cookbook, APEX uses two modern, super-intelligent translators:

CLIP: A translator that understands how images relate to language (great for "Does this look like a cat?").
DINOv2: A translator that understands textures, shapes, and fine details (great for "Does this fur look realistic?").
APEX uses these translators to understand the "flavor" of the image, then uses the "Shadow Puppet" trick to compare the generated image against real ones.

What the Paper Found (The Taste Test)

The authors put APEX to the test against the old judges (FID, KID) and the newer-but-still-rigid judges (CMMD). They tested them on:

Natural photos (like a park).
Faces (like a celebrity).
Medical scans (like an X-ray).
Satellite photos (like a view of a city from space).

The Results:

Stability: The old judges were like a shaky hand; if you showed them the same picture twice with a tiny change, they might give wildly different scores. APEX was steady and consistent, like a rock.
Fairness Across Domains: The old judges were great at grading faces but terrible at grading X-rays or satellite photos. APEX was equally good at judging all of them. It didn't get confused by the change in subject matter.
Sensitivity: When the AI chefs slowly improved their dishes (adding more detail step-by-step), APEX was the only one that noticed the tiny, subtle improvements at the very end. The old judges often got stuck or even thought the dish got worse when it actually got better.
Human Agreement: When real humans rated the images, APEX's scores matched human opinions almost perfectly, often better than the established "gold standard" (FID).

The Bottom Line

The paper claims that APEX is a better way to grade AI-generated images because it stops forcing images into old, rigid boxes. Instead, it uses modern, flexible tools to look at images from every possible angle, ensuring that the score reflects what the image actually looks like, whether it's a face, a tumor, or a satellite view. It's a more honest, stable, and human-aligned judge for the future of AI art.

Technical Summary: APEX (Assumption-free Projection-based Embedding eXamination)

1. Problem Statement

The evaluation of generative models has reached a critical juncture where traditional metrics struggle to keep pace with the unprecedented visual fidelity of modern synthesis. Current gold standards, such as the Fréchet Inception Distance (FID) and Kernel Inception Distance (KID), face two fundamental limitations:

Closed-Vocabulary Bottleneck: These metrics rely on feature extractors (e.g., Inception-v3) trained exclusively on ImageNet. Consequently, they lack the semantic richness and open-world alignment necessary to evaluate modern multimodal models or diverse domains like medical imaging and remote sensing.
Assumptive Bias: Existing metrics depend on rigid statistical formulations. FID assumes real and synthetic embeddings follow Gaussian distributions—a hypothesis empirically proven incorrect for high-dimensional, multimodal latent spaces. KID relies on Maximum Mean Discrepancy (MMD) with polynomial kernels, which introduces extreme sensitivity to kernel hyperparameter selection.

While recent alternatives have replaced Inception-v3 with foundation models (e.g., CLIP, DINOv2) to address the feature bottleneck, they often retain the problematic statistical assumptions (e.g., CMMD still uses MMD), leaving the field without a robust, assumption-free evaluation framework.

2. Methodology

The authors propose APEX (Assumption-free Projection-based Embedding eXamination), a framework designed to measure image quality by comparing distributions of embeddings without relying on parametric assumptions or kernel tuning.

Core Components

Feature Extraction: APEX is embedding-agnostic but primarily utilizes two state-of-the-art foundation models:
- CLIP (Contrastive Language–Image Pretraining): Provides high-level semantic alignment with natural language.
- DINOv2 (Knowledge DIstillation with NO labels): Offers fine-grained visual structure (texture, shape, spatial layout) via self-supervised learning.
- Implementation: APEX-CLIP uses 768-dimensional [CLS] tokens. APEX-DINO concatenates [CLS] tokens from layers 6, 12, and 23 of DINOv2 ViT-L/14 to form a 3,072-dimensional representation, capturing multi-scale visual abstractions.
Distance Metric (Sliced Wasserstein Distance - SWD): Instead of fitting parametric distributions or using kernel methods, APEX computes the SWD.
- Mechanism: SWD approximates the 2-Wasserstein distance by projecting high-dimensional distributions onto random 1D lines (directions $\theta$ ) and computing the Wasserstein distance on these projections.
- Advantages: It is mathematically grounded, assumption-free (no Gaussian or kernel assumptions), and computationally efficient ( $O(N \log N)$ ) compared to the quadratic cost of MMD.
Monte Carlo Approximation: The integral over the unit sphere is approximated by averaging $L$ random projections. The authors theoretically and empirically determine that $L=500$ provides a stable trade-off between estimation accuracy and computational cost.

Theoretical Guarantees

The paper provides a theoretical bound (Theorem 1) proving that the estimation error of the squared SWD is controlled by the number of projections $L$ . The bound depends on the intrinsic dimension $k$ of the manifold where embeddings reside and the manifold's diameter $D$ , but is independent of the ambient dimension $d$ . This justifies the scalability of SWD in high-dimensional spaces.

3. Key Contributions

Audit of Existing Metrics: A systematic analysis of failure modes in current metrics, highlighting their sensitivity to backbone choices, distributional assumptions, kernel configurations, and cross-domain shifts.
APEX Framework: Introduction of a projection-based evaluation framework that combines foundation models (CLIP/DINOv2) with SWD to eliminate rigid parametric assumptions and kernel hyperparameter dependencies.
Stability Analysis: Theoretical and empirical investigation into the number of projections required for stable SWD estimation, identifying $L=500$ as the optimal point for convergence.
Comprehensive Benchmarking: An extensive evaluation protocol spanning five heterogeneous domains (natural images, faces, dermoscopy, radiography, remote sensing), testing robustness to pixel/latent degradations, finite-sample stability, and generative refinement.
Human Alignment: A dedicated perceptual study demonstrating that APEX correlates strongly with human judgments, competitive with the strongest baselines.

4. Experimental Results

The evaluation compares APEX against FID, KID, CMMD, and DINO+MMD across multiple criteria:

Intra- and Cross-Dataset Stability:
- Intra-dataset: APEX variants exhibit low finite-sample bias and low coefficient of variation (CV $\le$ 0.06), outperforming KID and DINO+MMD (which suffer from high variance despite unbiased means) and FID (which relies on smooth Gaussian fits).
- Cross-dataset: APEX achieves the lowest cross-domain consistency error ( $\Lambda$ ), indicating that its degradation signals are more consistent across diverse domains (e.g., medical vs. natural images) compared to foundation-model-based baselines like CMMD.
Degradation Sensitivity: APEX demonstrates monotonic sensitivity to progressive perturbations (noise, blur, compression, latent distortions) across all domains. In contrast, baselines often show non-monotonic fluctuations or domain-specific failures.
Generative Refinement: During the denoising process of Stable Diffusion, APEX-CLIP shows the most significant drop in distance scores during the final refinement steps, indicating higher sensitivity to subtle perceptual improvements than FID or KID.
Human Correlation: In a study with 10,850 annotated images, APEX-CLIP achieved a Spearman $\rho$ of 0.844, nearly matching FID (0.847) and significantly outperforming CMMD (0.813) and DINO+MMD (0.651). This suggests the projection-based approach extracts more perceptually relevant information from the same embedding space.
Efficiency: While feature extraction dominates runtime, the distance computation for APEX scales as $O(N \log N)$ , bypassing the prohibitive $O(N^2)$ overhead of MMD-based methods.

5. Significance and Claims

The paper claims that APEX addresses the "closed-vocabulary" and "assumptive bias" bottlenecks that hinder current image quality assessment. By leveraging the Sliced Wasserstein Distance within the latent spaces of foundation models, APEX provides:

Robustness: Superior stability across diverse domains and under various image degradations.
Reliability: High correlation with human perception without requiring rigid statistical assumptions.
Scalability: Efficient computation suitable for high-dimensional embedding spaces.

The authors position APEX not as a replacement for all existing metrics, but as a principled, assumption-free alternative that offers a more stable and flexible baseline for evaluating the next generation of generative models. They acknowledge limitations, noting that performance still depends on the quality of the underlying embedding space and that the overall cost is dominated by feature extraction, particularly for the high-dimensional APEX-DINO variant. Future work is suggested to explore adaptive projection strategies and extensions to broader multimodal models.