💻 computer science

AI Outperforms Humans in Personalized Image Aesthetics Assessment via LLM-Based Interviews and Semantic Feature Extraction

This paper introduces an integrated deep learning and large language model system that outperforms human predictors and even an individual's own future self in assessing personalized image aesthetics by actively eliciting preferences through semi-structured interviews and leveraging both low-level and high-level semantic features.

Original authors: Yoshia Abe, Tatsuya Daikoku, Yasuo Kuniyoshi

Published 2026-05-15

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yoshia Abe, Tatsuya Daikoku, Yasuo Kuniyoshi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Can a Computer Know Your Taste Better Than You Do?

Imagine you have a friend who loves taking photos. You look at their photos and say, "I love this one!" and "I hate that one." But if you try to explain why you love the first one, you might struggle. You might say, "It's the lighting," or "The colors feel warm," but those are vague guesses.

This paper asks a big question: Can an AI figure out exactly what you like, better than you can explain it to yourself, or better than your friends can guess?

The researchers built a special AI system to answer this. They found that, surprisingly, the AI was better at predicting your personal taste than you were at predicting your own future taste, and definitely better than your friends guessing.

How the AI Works: The "Super-Interviewer"

Most old AI systems for judging photos are like a robotic art critic who only looks at the math. It checks if the image is bright, if the edges are sharp, or if the colors are balanced. It knows the "rules" of photography but doesn't understand you.

The new system in this paper is different. It's like a super-interviewer combined with a mathematician.

The Interview (The Chat):
Instead of just looking at your photos, the AI starts a conversation with you. It uses a "Large Language Model" (a very smart chatbot) to ask you deep, open-ended questions.
- Analogy: Imagine a detective who doesn't just ask, "Did you see the suspect?" but instead asks, "What kind of stories make you feel excited? Do you like old buildings or modern ones? Does a photo of a rainy street make you feel cozy or sad?"
- The AI asks follow-up questions based on your answers, digging deeper into your specific tastes, memories, and feelings.
The Translation (The Feature Extraction):
Once the AI understands your "vibe" from the chat, it goes back to looking at the photos. But this time, it doesn't just look at the math. It translates your chat answers into "high-level" features.
- Analogy: If you said in the chat, "I love photos that feel like a quiet Sunday morning," the AI learns to recognize that specific feeling in a picture. It turns your abstract feeling into a concrete data point it can use.
The Prediction (The Guess):
The AI combines the "math" (low-level features like brightness) with your "vibe" (high-level features from the chat) to predict how much you will like a new photo.

The Big Experiment: Who is the Best Judge?

The researchers tested this system with 30 people. They showed them 300 photos and asked them to rate them. Then, they tested four different "predictors" to see who could guess the ratings best for a new set of photos:

The Old AI: Just the math-based robot (no chat).
The Chatbot AI: An AI that tried to guess your taste using a few examples but didn't interview you deeply.
Your Friends: Real humans who were given your chat history and your past ratings, then asked to guess how you would rate new photos.
Your Future Self: You, coming back two weeks later to rate the same photos again.

The Results:

The New AI System won. It was the most accurate at guessing what you would like.
Your Friends lost. They were the worst at guessing. Why? Because they kept projecting their own tastes onto you. If they liked a photo, they assumed you would too. They couldn't separate their own brain from yours.
Your Future Self lost. Even you, two weeks later, didn't rate the photos the same way you did before. Your taste fluctuates. The AI, however, remembered your "snapshot" of taste from the interview perfectly and didn't change its mind.

The "High-Rated" Surprise

The AI was especially good at predicting which photos you would love (give a 5-star rating).

Analogy: It's like a music recommendation engine that is great at suggesting songs you might "kind of like," but this AI was amazing at finding the songs that would make you stop and say, "This is my favorite song ever." The researchers found that understanding your deep, personal story (via the interview) was the secret sauce for predicting these "love" ratings.

What Does This Mean? (According to the Paper)

The paper suggests a fascinating idea: At any single moment in time, an AI might understand your aesthetic soul better than you do.

Humans are messy: Our tastes change, and we are biased by our own opinions. We can't easily explain why we love something.
The AI is a mirror: By interviewing you and locking in your answers, the AI creates a perfect, unchanging "map" of your current preferences. It doesn't get tired, it doesn't get distracted, and it doesn't project its own feelings onto you.

The Caveats (What the Paper Says)

The researchers are careful to note a few limits:

The Group was Small: They only tested 30 people, all in their 20s and mostly male.
The Language: The interviews were in Japanese.
The Photos: They only used standard photographs, not paintings or abstract art.
The Cost: It takes a lot of computer power and money to run these interviews and calculations.

Summary

Think of this AI as a super-observer. It listens to your story, understands your unique "taste fingerprint," and uses that to predict what you'll love next. The study shows that this digital observer can be more consistent and accurate at understanding your personal taste than your friends can, and even more consistent than you are with yourself over time. It suggests that AI might eventually become a tool that helps us understand our own beauty standards better than we can on our own.

Technical Summary: AI Outperforms Humans in Personalized Image Aesthetics Assessment via LLM-Based Interviews and Semantic Feature Extraction

Problem Statement

Accurately predicting individual aesthetic preferences for images is a fundamental challenge in artificial intelligence. While deep learning (DL) models have advanced the field of Generic Image Aesthetics Assessment (GIAA)—predicting average population ratings—they struggle with Personalized Image Aesthetics Assessment (PIAA). Aesthetic evaluation is inherently subjective and dependent on individual internal processing, meaning population averages often discard critical individual-specific information.

Existing PIAA approaches face two primary limitations:

Data Scarcity: Collecting sufficient rating data for specific individuals is difficult.
Feature Limitations: Conventional methods rely heavily on objective, low-level physical features (e.g., color, texture) and fail to adequately capture high-level semantic features (e.g., narrative, emotion, cultural context) that drive subjective preference. Furthermore, these methods often lack a mechanism to actively elicit the specific, nuanced preferences of a target individual.

Methodology

The authors propose an integrated DL-LLM system designed to bridge the gap between objective image features and subjective human preference through active linguistic interaction. The system consists of two main subsystems:

1. The Interview System (Preference Elicitation)

To capture the target individual's aesthetic preferences, the system employs a multi-agent LLM framework to conduct a semi-structured interview.

Agents: Two agents operate asynchronously: an Interviewer that poses questions and an Analyzer that processes responses.
Mechanism: Unlike fixed-question surveys, the system uses a shared "Interview Data Container" to maintain context. The Analyzer generates insights and hypotheses from the user's responses, which the Interviewer uses to formulate the next question.
Themes: The interview covers three main themes: Preference Targets (subjects, stories, culture), Image-Evoked Reactions (emotional/physical responses), and Personal Tastes (general likes/dislikes).
Goal: To linguistically capture the underlying essence of the individual's aesthetic sensibility.

2. The Prediction System (Feature Extraction and Modeling)

This subsystem constructs a personalized predictive model by integrating low-level and high-level features.

Architecture:
- DL Module: A convolutional neural network (ResNet-50 backbone) pretrained on GIAA data and fine-tuned on the individual's data to provide an intermediate prediction based on low-level features.
- LLM Module: Uses the interview data to generate a set of high-level semantic features. It operates in two phases:
  - Feature Exploration: An iterative process where the LLM analyzes prediction errors from a temporary linear model and devises new linguistic feature descriptions (e.g., "evokes a sense of nostalgia"). The LLM then assigns an "applicability score" (0–1) to each image for these new features.
  - Model Training: A final Machine Learning (ML) model (Gradient Boosting Regressor) is trained using the extracted high-level features and the DL module's intermediate scores to predict the final aesthetic rating.
Training Data: The system uses a dataset of 300 images per participant (30 semantic categories × 3 rating classes × 20 images), split into training, validation, and test sets.

Experimental Setup

Participants: 30 individuals (all in their 20s; 25 male, 5 female).
Task: Participants rated 300 images on a 1.0–5.0 scale.
Baselines: The proposed system was compared against:
- DL-based Predictor: Standard fine-tuned ResNet-50.
- LLM-based Predictors: Various models (Claude, GPT, Gemini) using few-shot learning without the interview system.
- Human Predictors: Participants predicting the ratings of other anonymized participants (using their training data and interview summaries).
- Self-Retest: Participants re-rating the same images after a two-week interval (measuring within-person variability).
Metric: Mean Absolute Error (MAE) was used to measure prediction accuracy, allowing for direct comparison with within-person variability.

Key Results

Superior Performance: The proposed integrated system achieved the lowest MAE (0.549) among all predictors, significantly outperforming the DL-based predictor (0.561, $p=0.0328$ ).
High-Rated Image Accuracy: The system showed the most significant improvement over the DL baseline for images rated 5.0 (highly rated), with a mean improvement of 21%. This suggests high-level semantic features are crucial for capturing strong aesthetic appreciation.
Outperforming Human Predictors: Human predictors performed the worst among all groups. Their errors were systematically biased toward their own aesthetic values, indicating a difficulty in decoupling personal preference from predicting others'.
AI vs. Future Self: The proposed system outperformed the "Self-Retest" baseline. This indicates that an individual's aesthetic evaluation at a specific point in time is more accurately captured by the AI system than by their own future self, as human evaluation exhibits substantial temporal fluctuation.
User Experience: Participants reported high levels of self-insight and metacognition after the interview, suggesting the process helped them articulate their own preferences.

Significance and Claims

The paper claims that this work provides quantitative evidence that AI systems can capture individual aesthetic preferences more accurately than other humans or even an individual's future self.

Scientific Insight: The results suggest that human aesthetic values are subject to temporal fluctuation and are heavily biased by one's own value system when predicting others. In contrast, AI systems, once trained, exhibit temporal consistency and can flexibly adapt to new individuals without the systematic bias of personal projection.
Interpretation of Human Sensibility: The authors posit that AI may be better positioned to serve as a "deeper interpreter" of human aesthetic sensibility than humans themselves, as it can actively elicit and model subjective preferences without the interference of the predictor's own fluctuating values.
Technical Contribution: The work demonstrates a novel pipeline where LLMs are used not just for prediction, but for the active, iterative extraction of high-level semantic features from unstructured linguistic data to refine predictive models.

The paper concludes with a modest acknowledgment of limitations, including the small and demographically skewed participant pool, the reliance on Japanese language interviews, and the computational costs. It also notes the broader ethical implications regarding privacy and the potential for oversimplifying human values, urging future research into the irreducible uniqueness of human aesthetic experience.