🤖 machine learning

Banana100: Breaking NR-IQA Metrics by 100 Iterative Image Replications with Nano Banana Pro

The paper introduces Banana100, a dataset of 28,000 images degraded through 100 iterative editing steps, to reveal that multi-turn image editing causes severe quality accumulation and that current no-reference image quality assessment metrics fail to detect this degradation, posing significant risks to the stability of agentic AI systems.

Original authors: Kenan Tang, Praveen Arunshankar, Andong Hua, Anthony Yang, Yao Qin

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kenan Tang, Praveen Arunshankar, Andong Hua, Anthony Yang, Yao Qin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical photocopier that doesn't just copy a picture; it listens to your voice commands to change the picture. You tell it, "Add an apple," and it does. You tell it, "Make it brighter," and it does. This is how modern AI image editors work.

The paper "Banana100" asks a scary question: What happens if you keep using this photocopier over and over again?

Here is the breakdown of their discovery, explained simply.

1. The "Xerox Effect" (Iterative Degradation)

Imagine you take a crisp, high-definition photo of a banana.

Step 1: You ask the AI to "copy this exactly." It makes a copy. It looks 99% perfect.
Step 10: You ask it to copy the new copy. It looks a little grainy, like an old TV with static.
Step 20: You keep going. The image is now covered in digital snow, the colors are weird, and the banana looks like a blurry blob.

The researchers found that AI models are terrible at "copying" their own work. Every time they touch an image, they introduce tiny, invisible errors. When you feed the result back into the machine for the next round, those tiny errors pile up like dust in a corner. After about 10 to 20 rounds, the image is a mess of noise and distortion.

2. The "Broken Thermostat" (The Evaluator Failure)

Here is where it gets really weird. Usually, when an image gets ugly, we have tools (metrics) to measure how bad it is. These tools act like a thermostat: if the room gets hot (the image gets bad), the thermostat should scream "Too hot!"

The researchers tested 21 of these "thermostats" (called NR-IQA metrics). They expected the tools to say, "Hey, this image is garbage now!"

Instead, the tools said, "Wow, this is beautiful!"

The Analogy: Imagine you take a pristine, clean white shirt and start rubbing it in mud. After 20 minutes, it's a brown, muddy mess. But your "Cleanliness Meter" suddenly beeps and says, "This is the cleanest shirt I've ever seen!"
The Reality: The AI tools were so confused by the specific type of "digital mud" the generator created that they thought the noise was actually a sign of high quality. In one famous example, a clean image got a "bad" score, but after 20 rounds of corruption, the same image got a "perfect" score.

3. The "Clueless Artist" (Instruction Failure)

Not only did the images get ugly, but the AI also started forgetting what it was supposed to do.

The Counting Fail: If you asked the AI to "add one apple" 10 times, it might add 10 apples in the first round, then forget to add any in the next, or suddenly add a whole orchard.
The Hallucination: The AI would look at its own messy, noisy output and write a report saying, "I did a great job! The details are perfect!" even though the image was covered in static. It was lying to itself.

4. The "Digital Snowball" (Why This Matters)

Why should we care?

The Training Loop: If AI models are trained on data created by other AI models, and that data is full of this invisible "digital snow," the next generation of AI will learn from garbage. This is called Model Collapse. It's like a student studying from a textbook that has been photocopied 100 times; eventually, the text becomes unreadable gibberish.
The Silent Poison: Because the "thermostats" (quality checkers) are broken, this garbage data is slipping into our databases. We are unknowingly feeding our future AI systems a diet of noise.

5. The Solution: Banana100

The authors created a dataset called Banana100.

They took 13 different images (a forest, a peacock, a plate of dumplings).
They asked an AI to edit them 100 times in a row.
They saved all 28,000 resulting images.

This dataset is like a "crash test dummy" for the AI world. It proves that current AI editors are fragile and current quality checkers are blind. By releasing this data, they hope scientists can build:

Better Editors: AI that doesn't get messy when it copies itself.
Better Checkers: Tools that can actually tell the difference between a clean image and a noisy one.

The Bottom Line

The paper warns us that while AI is amazing at making pictures, it is currently fragile when asked to keep editing them. It's like a game of "Telephone" played by a robot: the message (the image) gets distorted with every turn, and the robot's own quality control system is too broken to notice the distortion until it's too late.

1. Problem Statement

The paper addresses a critical, underexplored failure mode in multi-modal agentic systems and iterative image editing: iterative degradation.

The Phenomenon: While modern image editing models (e.g., Nano Banana Pro) perform well in single-turn edits, repeated multi-turn editing causes minor, imperceptible artifacts to accumulate. Over time (typically 5–10 steps), this leads to severe visible noise (static, color tints, scatter points), distortion of high-frequency details, and a failure to follow simple editing instructions.
The Evaluation Gap: Current No-Reference Image Quality Assessment (NR-IQA) metrics, which are widely used to filter synthetic data and evaluate agentic pipelines, fail to detect this degradation. Alarmingly, many metrics assign better (lower or higher, depending on the metric) scores to heavily degraded, noisy images than to clean originals.
The Risk: If low-quality synthetic data generated by multi-turn edits escapes quality filters, it could lead to model collapse in future training cycles and exacerbate the fragility of deployed agentic systems.

2. Methodology

The authors introduced Banana100, a comprehensive dataset and evaluation framework designed to systematically study these failures.

Dataset Construction (Banana100):
- Source: 13 diverse initial images (11 photorealistic, 2 anime-style) generated by AI, covering various textures (e.g., grids, food, landscapes, characters).
- Process: Each image underwent 100 iterative editing steps using Nano Banana Pro.
- Prompts: The primary task was "exact replication" (producing an exact replica with no alterations) to isolate quality degradation from semantic changes. Variants included multi-step reconstruction, object addition, and hyperparameter sweeps (seed, temperature, resolution).
- Scale: The dataset contains 28,000 images generated at a cost of ~$4,000.
Evaluation Protocol:
- Visual Analysis: Qualitative assessment of failure modes at three levels: sub-object, object, and image.
- Metric Testing: The authors evaluated 21 popular NR-IQA metrics (including BRISQUE, NIQE, CLIPIQA, MUSIQ, etc.) against the dataset.
- Scoring Definition: They defined a normalized score gap ( $\Delta_i$ ) to measure the trend of quality scores from Step 1 to Steps 5, 10, and 20. A successful metric should show a negative gap (indicating quality drop).
- Self-Evaluation Check: They used an LLM-as-a-judge to analyze the model's internal "reasoning summaries" to see if the generator acknowledged its own noise.

3. Key Contributions

Banana100 Dataset: A large-scale, controlled dataset of 28,000 images demonstrating progressive quality degradation through 100 editing steps. It is the first dataset to systematically capture the "noise accumulation" phenomenon in iterative editing.
Failure Mode Taxonomy: The paper categorizes instruction-following failures into three distinct levels:
- Sub-Object Level: Simplification of details (e.g., eye colors in a character grid merging into fewer colors).
- Object Level: Failure to add objects correctly (e.g., counting errors, replacing objects instead of adding them) and inconsistent background degradation.
- Image Level: Aspect ratio mismatches, persistent noise that resists denoising prompts, and failure of "round-trip" editing (e.g., mirroring/rotation) to restore quality.
Exposure of NR-IQA Flaws: The study quantitatively proves that 21 out of 23 tested NR-IQA metrics are fundamentally flawed for detecting model-induced degradation. They often assign better scores to noisy images than clean ones, rendering them unreliable for filtering synthetic data.

4. Key Results

Metric Failure: None of the 21 traditional NR-IQA metrics (based on CNNs or hand-crafted features) consistently identified degradation. In many cases (e.g., BRISQUE), the score for a noisy image after 20 replications was significantly "better" than the clean original (e.g., dropping from 34.1 to -9.8).
VLM-Based Metrics: Only two recent metrics based on Large Vision-Language Models (RALI and VisualQuality-R1) successfully detected the degradation with 0 failure cases, though they showed some sensitivity to semantic content changes.
Generator Blindness: The model's internal reasoning summaries exhibited "hallucinated confidence," claiming the output aligned with the prompt even when severe degradation occurred. The model only acknowledged noise in its reasoning summary after ~20–37 steps, long after the degradation was visually obvious to humans (5–10 steps).
Generalizability: The degradation phenomenon was confirmed in other models (Nano Banana 2 Fast, FLUX.2, Qwen Image Edit), though the specific noise patterns (e.g., wrinkles vs. scatter points) varied. The same NR-IQA metrics failed across these different models.

5. Significance and Implications

Data Pollution Warning: The findings suggest that current quality filters are ineffective. As AI-generated content proliferates, future training datasets may be silently polluted with degraded, noisy images, accelerating model collapse.
Agentic System Fragility: Autonomous agents relying on iterative refinement are inherently fragile. Without robust evaluators, these systems cannot maintain quality over long horizons.
Call to Action: The paper urges the community to:
1. Stop relying on traditional NR-IQA metrics for AI-generated content evaluation.
2. Develop new metrics specifically trained to detect model-induced noise patterns rather than just heuristic distortions (blur, compression).
3. Create more robust image editing models that prevent artifact accumulation during iterative processes.

The authors have released the full code and the Banana100 dataset to facilitate the development of more robust models and evaluation standards.