🤖 machine learning

Panoptic Pairwise Distortion Graph

This paper introduces the Panoptic Pairwise Distortion Graph (DG), a novel framework that represents image pairs as structured region-level graphs to enable fine-grained comparative image assessment, supported by the PandaSet dataset, PandaBench benchmark, and the Panda architecture.

Original authors: Muhammad Kamran Janjua, Abdul Wahab, Bahador Rashidi

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammad Kamran Janjua, Abdul Wahab, Bahador Rashidi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a professional art critic tasked with comparing two photos of the same scene: one is a pristine, high-definition original, and the other is a version that has been damaged by rain, smudged by a finger, or compressed into a low-quality JPEG.

Most current AI systems act like a blurry-eyed tourist. They look at the whole picture and say, "Hmm, the second one looks a bit worse overall." They give you a single score for the entire image, missing the specific details. They might tell you the sky looks bad, when in reality, the sky is perfect, but the person's face is blurry.

This paper introduces a new way for AI to look at images, called the Distortion Graph (DG). Here is the breakdown of their invention, the PANDA system, using simple analogies.

1. The Problem: The "Whole Picture" Blind Spot

Current AI models (like the fancy chatbots you use today) are great at talking about images, but they are terrible at spotting specific flaws.

The Analogy: Imagine a doctor looking at a patient's X-ray. Instead of checking the heart, lungs, and bones individually, the doctor just says, "The patient looks sick." They miss the fact that the lungs are fine, but the broken leg is the real issue.
The Paper's Finding: Even the smartest AI models fail when you ask them, "Which part of this image is blurry?" or "Is the sky in Image A better than Image B?" They tend to guess or give generic answers.

2. The Solution: The "Distortion Graph" (DG)

The authors propose a new way to represent an image pair. Instead of treating the image as one big blob, they break it down into a structured map, like a subway map or a family tree.

The Nodes (Stations): Every object in the image (a person, a tree, a car) is a "station" on the map.
The Edges (Tracks): The lines connecting the stations in Image A to the stations in Image B represent the comparison.
The Labels (Signs): On every track, there is a sign that says exactly what happened:
- Type: "Blur" or "Noise."
- Severity: "Mild" or "Severe."
- Score: "8/10" or "2/10."

The Metaphor: Think of the Distortion Graph as a detective's evidence board. Instead of just saying "The suspect is guilty," the board has photos of specific clues (the muddy shoe, the torn shirt) with notes on exactly how they were damaged and how they compare to the original evidence.

3. The Toolkit: PANDASET, PANDABENCH, and PANDA

To teach AI to build these "detective boards," the authors created three things:

PANDASET (The Training School): A massive library of 500,000+ image pairs. They took clean photos and artificially "ruined" them in 15 different ways (rain, blur, pixelation, darkness) at different levels of severity. Crucially, they labeled exactly which part of the photo was ruined.
PANDABENCH (The Final Exam): A test with three levels of difficulty:
- Easy: The whole image is slightly blurry.
- Medium: The sky is clear, but the person is blurry.
- Hard: The sky is noisy, the person is blurry, and the car is pixelated.
- Result: Even the most advanced AI models (like GPT-4 or Gemini) failed miserably on the "Hard" exam, often getting scores close to random guessing.
PANDA (The Student): A new, efficient AI model designed specifically to look at the image pair and draw the "Distortion Graph." It doesn't just guess; it systematically analyzes every region and outputs a structured map of the damage.

4. The "Superpower" Trick

The most exciting part of the paper is what happens when you combine the new PANDA model with the old, smart AI models.

The Analogy: Imagine you have a brilliant but lazy professor (the big AI model) who is bad at math. You give them a calculator (PANDA) that has already done the math and written down the answer on a sticky note.
The Result: When you show the professor the sticky note (the Distortion Graph) and ask, "So, what do you think?", the professor suddenly becomes much smarter. They use the structured notes to give a much better, more accurate explanation.
Why it works: The graph acts as a "crutch" or a "hint" that forces the AI to focus on the specific details rather than hallucinating a generic answer.

Summary

This paper argues that to truly understand image quality, we need to stop looking at the "forest" and start mapping the "trees."

Old Way: "This picture is bad."
New Way (Distortion Graph): "The sky is perfect, the tree is slightly blurry, but the person's face is severely pixelated."

By teaching AI to build these detailed maps, the authors have created a system that can not only spot specific flaws but also help even the smartest AI models understand images with human-like precision. It's a shift from guessing to diagnosing.

1. Problem Statement

Current approaches to Image Quality Assessment (IQA) and comparative image analysis predominantly rely on holistic, top-down processing. Existing Multimodal Large Language Models (MLLMs) and specialized IQA models typically analyze an entire image as a single unit or rely on implicit region-level understanding without explicit structural grounding.

This "global view" approach suffers from several critical limitations:

Lack of Fine-Grained Understanding: Models struggle to identify specific distortions (e.g., blur, noise) affecting specific objects (e.g., a person's face vs. the background) within a single image.
Inability to Perform Structured Comparison: When comparing two images, current models often fail to provide a systematic, region-by-region comparison, leading to hallucinations, missed regions, or rigid template responses.
Context Limitations: MLLMs often fail to reason over variable numbers of regions due to context window constraints and a lack of structured representation for pairwise relationships.

The paper argues that perceptual decision-making requires a structured, region-grounded representation that explicitly models the relationships and attributes of image regions across a pair of images.

2. Methodology

The authors propose a novel framework centered around the Distortion Graph (DG) and a specialized architecture called PANDA.

A. The Distortion Graph (DG)

The DG is a structured topological representation of an image pair, defined as a 4-tuple $G = (O_{IA}, O_{IT}, E_D, E_S)$ :

Nodes ( $O_{IA}, O_{IT}$ ): Represent panoptic regions (objects) in the Anchor ( $I_A$ ) and Target ( $I_T$ ) images.
Distortion Edges ( $E_D$ ): Directed edges connecting matched regions between the two images. These encode:
- Comparative Relations: Predicates indicating which region is better/worse (e.g., "significantly better," "same").
- Distortion Attributes: The specific type of degradation (e.g., blur, noise, rain) and its severity (minor, moderate, severe).
- Quality Scores: A numerical score indicating the quality of the region.
Scene Edges ( $E_S$ ): Optional edges capturing intra-image semantic relationships (though the primary focus is on distortion).

The DG enforces three properties: Validity (edges only connect matched regions), Ordering (edges are directed from Anchor to Target), and Functional Comparison (exactly one relation exists per matched pair).

B. The PANDA Architecture

To learn the Distortion Graph, the authors introduce PANDA, an efficient neural network architecture:

Input Processing: Takes a pair of images (Anchor and Target).
Feature Extraction: Uses a pretrained encoder (e.g., DINOv2) to extract global features and a panoptic segmentation model (e.g., SAM) to generate binary masks for regions.
Token Pooling: Introduces learnable token vectors associated with each region mask to bridge the gap between image features and specific regions.
Degradation Decoder:
- Utilizes a Transformer-based decoder with Self-Attention (within an image) and Cross-Attention (between the Anchor and Target images).
- This allows each region in one image to attend to its corresponding region in the other image to learn comparative relationships.
Prediction Heads: Four parallel heads predict:
- Relation Head: The comparative predicate (e.g., "better," "worse").
- Distortion Head: The type of distortion.
- Severity Head: The level of degradation.
- Score Head: A regression head for the quality score.

3. Key Contributions

The paper makes three primary contributions to the field of low-level vision and MLLMs:

PANDASET (Dataset):
- A large-scale region-level dataset containing 528K image pairs.
- Built from PSG and Seagull-100w, covering 15 distortion types (including weather, camera, and digital distortions) with 4 severity levels.
- Includes deterministic region-level quality scores and comparative labels derived from the TOPIQ metric.
PANDABENCH (Benchmark):
- A systematic evaluation suite with three difficulty splits:
  - Easy: Uniform distortion type across the image.
  - Medium: One image has uniform distortion; the other has mixed distortions.
  - Hard: Both images have mixed distortions and severities across different regions.
- Evaluates models on four tasks: Comparison, Distortion Type, Severity, and Quality Score.
PANDA (Architecture):
- An efficient, region-first architecture designed specifically to generate Distortion Graphs, outperforming general-purpose MLLMs in structured pairwise assessment.

4. Experimental Results

The authors evaluated PANDA against state-of-the-art open-source MLLMs (e.g., Q-Insight, DepictQA, Gemma-3) and closed-source frontier models (e.g., GPT-4o, GPT-5 Mini, Gemini 2.5 Pro).

Performance Gap: Current MLLMs, even frontier models, perform poorly on region-level comparative tasks. In the Hard split, many models degrade to random chance levels (e.g., <20% accuracy) for distortion and severity classification.
PANDA Superiority: PANDA achieves state-of-the-art performance across all splits.
- Easy Split: ~58% Accuracy (Comparison), ~78% (Distortion).
- Hard Split: ~40% Accuracy (Comparison), ~27% (Distortion). While performance drops on Hard, it remains significantly higher than baselines (which often drop below 15%).
Emergent Capabilities (Chain-of-Thought): A key finding is that feeding the predicted Distortion Graph as a "hint" into a Chain-of-Thought (CoT) prompt for GPT-5 Mini improves its region-wise understanding by approximately 15%. The model learns to use the graph as a structured scaffold, overriding incorrect graph predictions when visual evidence contradicts them.
Efficiency: PANDA is computationally efficient (inference time ~3.5s per pair) compared to querying MLLMs multiple times per region (which can take minutes or hours).

5. Significance and Impact

Paradigm Shift: The paper moves the field from "global image scoring" to "structured, region-grounded pairwise reasoning." It demonstrates that decomposing perceptual decisions into atomic region relationships yields more interpretable and accurate results.
Bridging MLLMs and Low-Level Vision: It provides a mechanism to unlock the fine-grained reasoning capabilities of MLLMs. By using DG as an intermediate structured representation, MLLMs can overcome their tendency to hallucinate or miss details in complex comparative tasks.
Generalizability: The Distortion Graph formalism is not limited to IQA; the authors suggest it can be applied to video action differencing, forgery detection, and medical image comparison, offering a unified framework for comparative visual reasoning.
Resource Availability: The release of PANDASET and PANDABENCH fills a critical gap in the literature, providing the first large-scale benchmark for region-level pairwise distortion analysis.

In conclusion, this work establishes that structure is essential for fine-grained visual assessment. By introducing the Distortion Graph, the authors provide a robust data structure and learning framework that enables machines to understand where and how images differ, a capability that current top-down MLLMs lack.