Fine-grained Image Aesthetic Assessment: Learning Discriminative Scores from Relative Ranks

This paper introduces FGAesthetics, a large-scale fine-grained image aesthetic assessment database with pairwise comparison annotations, and proposes FGAesQ, a novel framework that leverages relative ranks through specialized tokenization and alignment techniques to achieve superior discriminative performance in both fine-grained and coarse-grained aesthetic evaluation scenarios.

The Gist

Imagine you are a professional photo editor. You have just taken 50 photos of the same sunset. To a casual observer, they all look "beautiful." But to you, the expert, one has the perfect golden glow, another has a slightly better composition, and a third is just "okay."

The Problem:
Current computer programs that judge photo beauty are like a very strict teacher who only knows the difference between a "masterpiece" and a "scribble." They can easily tell you that a sunset photo is better than a photo of a blurry sock. But if you ask them to pick the best sunset among 50 very similar sunsets, they get confused. They can't see the tiny, subtle differences that make one photo "great" and another just "good."

The Solution:
This paper introduces a new system called FGAesQ and a massive new training library called FGAesthetics. Think of it as upgrading the computer from a "Pass/Fail" grader to a "Fine-Tuned Art Critic."

Here is how they did it, broken down into simple concepts:

1. The New Training Library (FGAesthetics)

To teach the computer to be a fine-tuned critic, the researchers didn't just show it random photos. They created a special "training gym" with 32,000 photos organized into series.

• The Analogy: Imagine you are training a wine taster. You don't just give them cheap wine and expensive wine. You give them five bottles of the same vintage from the same vineyard, where the only difference is a tiny hint of oak in one and a hint of berry in another.
• The Data: They collected photos from three sources:
  • Natural: Real photos taken by humans (like burst shots from a camera).
  • AIGC: AI-generated images (where the AI tries to make the same picture 10 times with slight variations).
  • Cropping: The same photo cropped in different ways (like zooming in slightly on a face).
• The Labeling: Instead of asking humans "Rate this photo 1 to 10," they asked: "Between Photo A and Photo B, which one is slightly better?" This is much easier for humans to do accurately and creates a "ranking" rather than a vague score.

2. The New Brain (FGAesQ)

Once they had the training data, they built a new AI model. This model uses three clever tricks to learn the subtle differences:

Trick A: The "Spotlight" (Difference-preserved Tokenization)

• The Problem: When looking at two very similar photos, the computer wastes energy looking at the parts that are exactly the same (like the blue sky).
• The Fix: The model acts like a detective with a flashlight. It zooms in only on the tiny spots where the photos are different (e.g., a slightly brighter highlight on a nose, or a slightly straighter horizon line). It ignores the boring, identical parts to focus its brainpower on the details that matter.

Trick B: The "Art Critic's Voice" (Comparative Text-assisted Alignment)

• The Problem: Sometimes pixels alone aren't enough to explain why one photo is better.
• The Fix: The researchers used a super-smart AI (like GPT-4) to write short, snappy comparisons. Instead of just seeing the images, the model "reads" a note that says: "Image A has a warmer, more inviting light, while Image B feels a bit cold and distant." This helps the model connect the visual pixels to the feeling of the image.

Trick C: The "Race Track" (Rank-aware Regression)

• The Problem: In the old days, models tried to guess a specific number (e.g., "This photo is a 7.4"). But in fine-grained assessment, the exact number matters less than the order.
• The Fix: The model stops trying to guess the exact score. Instead, it learns to run a race. It asks, "If I line these 5 photos up from best to worst, am I getting the order right?" It learns that being "slightly better" is a specific relationship, not just a random number.

3. The Results

The researchers tested their new "Fine-Tuned Art Critic" against the old "Pass/Fail" models.

• The Old Models: They were great at saying "This is a good photo" vs. "This is a bad photo." But when asked to rank 10 similar photos, they often got the order wrong.
• FGAesQ: It nailed the rankings. It could tell you exactly which of the 50 sunset photos was the winner.
• The Best Part: Even though it learned to be a fine-tuned critic, it didn't forget how to be a general critic. It's still great at telling the difference between a masterpiece and a scribble. It's a "Jack of all trades, master of the subtle."

Why Does This Matter?

This technology is a game-changer for:

• Social Media: Automatically picking the absolute best photo from your camera roll to post, rather than just a "good" one.
• AI Art: Helping AI generators tweak their output to get that one perfect version of an image.
• Photo Albums: Organizing your vacation photos so the most beautiful ones are at the top, not just the first ones you took.

In short, this paper teaches computers to stop just "seeing" pictures and start truly "feeling" the subtle nuances that make a photo great.

Technical Summary

1. Problem Definition

The paper addresses a critical gap in Image Aesthetic Assessment (IAA): the inability of current state-of-the-art models to distinguish subtle aesthetic differences between visually similar images.

• Context: Existing benchmarks (e.g., AVA, TAD66K) focus on coarse-grained evaluation, where images with significant aesthetic differences are scored independently on an absolute scale.
• The Challenge: In real-world scenarios (e.g., selecting the best photo from a burst sequence, refining AI-generated outputs, or choosing optimal crops), images share the same semantic content but differ in nuanced aesthetic details (lighting, composition, color harmony).
• Key Obstacles:
  1. Semantic Interference: Strong semantic similarity in image series confuses deep models pre-trained for semantic tasks, making it hard to extract aesthetic differences.
  2. Subtle Variations: Minor changes require highly discriminative representations that standard regression models often miss.

2. Methodology

The authors propose a two-pronged solution: a new benchmark dataset (FGAesthetics) and a novel framework (FGAesQ).

A. FGAesthetics Dataset

A large-scale, fine-grained IAA database designed specifically for relative ranking tasks.

• Scale: 32,217 images organized into 10,028 series.
• Sources: Diverse categories including Natural (burst photos, video frames), AIGC (text-to-image variations), and Cropping (different aspect ratios/compositions of the same image).
• Annotation Protocol:
  • Series Refinement: A "Metrics-MLLMs-Human" protocol filters data to ensure series are visually similar yet distinguishable. This involves SSIM/IoU metrics, MLLM (Gemini-2.5-pro) coherence checks, and human qualification.
  • Rank Calibration: Instead of absolute scoring, annotators perform pairwise comparisons within each series. Ambiguous pairs are filtered, and global rankings are derived using the Bradley-Terry model logic.

B. FGAesQ Framework

A novel IAA model that learns discriminative scores from relative ranks while maintaining coarse-grained capabilities. It consists of three core modules:

1. Difference-preserved Tokenization (DiffToken):

   • Goal: Address semantic interference by focusing computational resources on regions where aesthetic differences actually exist.
   • Mechanism: For a pair of similar images, the model calculates patch-level similarity (SSIM). Patches with low similarity (aesthetic-decisive regions) are kept at original resolution, while homogeneous regions are downsampled. This preserves fine details crucial for discrimination without increasing token count excessively.

2. Comparative Text-assisted Alignment (CTAlign):

   • Goal: Enhance visual representation discrimination using semantic context.
   • Mechanism: Uses an MLLM (GPT-4o) to generate comparative textual descriptions (e.g., "Image A is more vibrant but lacks depth compared to B") for image pairs. The model aligns the visual embedding difference ($E_v(x) - E_v(y)$) with the text embedding ($E_t(T)$) via a contrastive loss, forcing the visual encoder to learn features that match the textual rationale.

3. Rank-aware Regression (RankReg):

   • Goal: Calibrate absolute scores using relative ranking information.
   • Mechanism: The model predicts absolute scores but optimizes them using a ListMLE loss based on the Bradley-Terry model. This ensures that the predicted scores strictly adhere to the ground-truth ranking order of the series, refining the regression space for subtle distinctions.

Training Strategy: A two-stage approach. First, pre-training on coarse-grained data (AVA) to establish foundational aesthetic perception. Second, joint learning alternating between coarse-grained (absolute score) and fine-grained (relative rank) batches to refine the model's discriminative power.

3. Key Contributions

1. Definition of FG-IAA: Formally defines the task of Fine-grained Image Aesthetic Assessment and identifies the limitations of current absolute-scale models.
2. FGAesthetics Benchmark: Introduces the first large-scale dataset with 10,028 series and pairwise ranking labels across Natural, AIGC, and Cropping domains.
3. FGAesQ Model: Proposes a unified framework that achieves superior fine-grained discrimination without sacrificing coarse-grained performance, utilizing DiffToken, CTAlign, and RankReg.

4. Experimental Results

The paper presents extensive experiments on FGAesthetics and cross-dataset validation.

• Performance on FGAesthetics:
  • FGAesQ outperforms all state-of-the-art methods (including NIMA, MUSIQ, VILA, Q-Align, and Charm) in both Pair-level (local discrimination) and Series-level (global ranking) metrics.
  • Notably, FGAesQ achieves a 0.753 s-Acc and 0.600 s-SRCC on the series level, significantly beating the previous best (Charm at 0.477 s-SRCC).
• Coarse-Grained Robustness:
  • Unlike other models that degrade significantly on coarse-grained tasks when fine-tuned for ranking, FGAesQ maintains competitive performance on the AVA benchmark (SRCC: 0.770, PLCC: 0.781).
• Generalization:
  • FGAesQ shows superior generalization on other benchmarks (ICAA17K, AADB, TAD66K), particularly in aesthetic attribute recognition (AADB), suggesting that fine-grained training enhances feature sensitivity.
• Ablation Studies:
  • Removing DiffToken drops performance significantly, proving the necessity of focusing on difference regions.
  • Removing CTAlign confirms the value of MLLM-generated comparative text.
  • Removing RankReg shows that direct ranking training is less effective than the proposed joint optimization.

5. Significance

• Paradigm Shift: Moves the field from "absolute scoring" to "relative discrimination," which is more aligned with practical applications like photo selection and AI generation refinement.
• Bridging the Gap: Demonstrates that a model can be both highly sensitive to subtle differences (fine-grained) and robust for general assessment (coarse-grained), solving the trade-off that plagued previous approaches.
• Practical Impact: The proposed methods and dataset provide a foundation for improving automated photo album management, AI image generation feedback loops, and smart photography tools where choosing the "best" among similar options is critical.