🤖 AI

In Search of the Ingredients of Open-Endedness: Replicating Picbreeder with Large Vision-Language Models

This paper investigates whether large Vision-Language Models can replicate the open-ended, creative discovery observed in human-driven Picbreeder experiments, finding that while VLMs produce distinct outputs, incorporating mechanisms like exploratory noise, behavioral diversity, and memory is necessary to better understand and potentially bridge the gap in generative capacity.

Original authors: Sam Earle, Kay Arulkumaran, Andrew Dai, Akarsh Kumar, Julian Togelius, Sebastian Risi

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sam Earle, Kay Arulkumaran, Andrew Dai, Akarsh Kumar, Julian Togelius, Sebastian Risi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a digital art gallery called Picbreeder. For years, real humans have visited this gallery to play a game of "evolutionary art." They start with a random, blurry blob of pixels. They pick the one they like best, ask the computer to make slight, random changes to it (like a genetic mutation), and then pick the best new version. They repeat this process over and over, creating a family tree of images. Over time, humans discovered they could evolve these blobs into recognizable things like cars, faces, or mushrooms, often stumbling upon these shapes by accident rather than by planning them.

The big question this paper asks is: Can an AI do this on its own?

The researchers replaced the human players with Large Vision-Language Models (VLMs)—super-smart AI that can see pictures and understand language. They wanted to see if these AIs could replicate the human ability to "open-endedly" discover new, interesting things without a specific goal in mind.

Here is what they found, explained through simple analogies:

1. The Setup: The AI Art Class

Think of the AI agents as students in an art class.

The Canvas: Instead of paint, they are evolving tiny neural networks (mathematical recipes) that generate images.
The Teacher: The AI is told, "Look at these images, pick the ones you like, and make new ones based on them."
The Goal: There is no "correct" answer. The goal is just to keep making new, interesting things forever.

2. The Problem: The AI Gets Stuck in a Rut

When the researchers let the AI play alone without any special tricks, it acted a bit like a student who gets bored and starts drawing the same thing over and over.

The "Fox" Trap: Instead of discovering a wide variety of animals, the AI would get stuck in a loop, creating dozens of slightly different versions of the same fox-like or fish-bone-like shape. It lacked the human ability to take a "bold leap" to a completely new idea.
The Memory Issue: Humans remember what they've seen before and try to avoid repeating themselves. The AI, however, struggled with memory.
- If the AI had no memory of its past moves, it would just spam the same image repeatedly.
- If the AI had too much memory (reading its entire history), it got overwhelmed and confused, often producing messy, noisy static instead of clear pictures.
- The Sweet Spot: The AI worked best when it remembered just the very last thing it did. This was enough to stop it from repeating itself immediately, but not so much that it got confused.

3. The Solutions: How to Make the AI More Creative

The researchers tried three main "ingredients" to see if they could spark more creativity, similar to how a teacher might try to motivate a student.

A. Adding a Little "Chaos" (Exploratory Noise)
Imagine a student who is too perfect at following rules. Sometimes, you have to tell them, "Hey, just pick a random color for once!"

The researchers added a small amount of randomness to the AI's choices. Instead of always picking the "best" image, the AI would occasionally pick a random one just to see what happens.
Result: This helped the AI escape the "fox trap" and find more diverse shapes. However, if they added too much chaos, the AI just started generating garbage noise. A little bit of randomness was the key to finding new paths.

B. Giving the AI "Personalities"
Imagine a classroom with 1,000 students, but instead of them all being the same, you give each one a unique personality trait. One is obsessed with "the smell of campfire smoke," another is "looking for the visual sound of a siren," and another is "bored and wants to break the rules."

The researchers created 1,000 different AI agents, each with a unique, quirky personality prompt.
Result: This created a lot of diversity. Because each agent was looking for something different, the gallery filled up with a huge variety of images.
The Catch: While the gallery was diverse, it also filled up with "adversarial" images—strange, psychedelic, noisy patterns that looked like TV static. These were images that satisfied the weird, abstract personalities of the agents but didn't look like anything real. It was like having a student who decided that "bad TV reception" was the most beautiful art form.

4. The Verdict: Humans Still Have the Edge

The paper concludes that while AI can play this game, it doesn't quite match the human version yet.

Human Gallery: Full of refined, recognizable objects (cars, faces, tools) that feel meaningful and evolved.
AI Gallery: Full of interesting patterns, but often stuck in loops, or filled with noisy, abstract static. The AI struggles to make the "big leaps" in creativity that humans do naturally.

The Takeaway:
The researchers didn't just build a better art bot; they built a laboratory to understand why humans are so good at open-ended discovery. They found that to get an AI to be truly creative, you need a delicate balance:

Just enough memory to avoid repeating yourself.
Just enough randomness to try new things.
A diverse group of "personalities" to explore different corners of the creative space.

But even with these ingredients, the AI still produces a lot of "noise" and hasn't quite mastered the art of finding meaningful, recognizable objects on its own. The paper suggests that we are still searching for the secret "ingredients" that make human creativity so special.

Technical Summary: In Search of the Ingredients of Open-Endedness: Replicating Picbreeder with Large Vision-Language Models

Problem Statement

The paper addresses the challenge of achieving artificial open-endedness: the capacity of an autonomous system to generate a seemingly endless supply of novel and meaningful artifacts without human intervention or predefined optimization targets. While machine learning relies heavily on static datasets, which are becoming a bottleneck, open-ended systems promise continuous generation of novelty. However, fully automated open-endedness remains a "grand challenge."

The authors posit that Picbreeder, a historical human-in-the-loop system where users collaboratively evolved images via neural networks (CPPNs), serves as a canonical exemplar of open-ended discovery. The core research question is whether Large Vision-Language Models (VLMs) can replicate the open-ended behavior of human users in this environment, and if so, what design ingredients (memory, exploration, diversity) are necessary to facilitate it.

Methodology

The authors implemented a fully artificial recreation of Picbreeder, replacing human users with frontier VLMs.

System Architecture

Underlying Engine: The system uses NEAT (NeuroEvolution of Augmenting Topologies) to evolve CPPNs (Compositional Pattern Producing Networks). Each CPPN takes $(x, y, r)$ coordinates as input and outputs hue, saturation, and brightness, generating $128 \times 128$ images.
Agent Loop: A "session" consists of an agent branching from an existing image or starting from a random population. The agent selects parents from a population of 15 images, applies mutations (and crossover), and evolves the population for 20 generations. At the final generation, the agent selects one image to publish to a shared archive and titles it.
Agent Interface: Agents receive a sample of the archive (categorized by top-rated, best new, most branched, latest, and random) and are prompted to select parents, toggle color/structure modes, and adjust mutation strength. They must provide a rationale for their decisions.

Experimental Interventions

To identify the "ingredients" of open-endedness, the authors varied three key parameters:

Memory (Context Length, $CL$): The number of previous interaction steps included in the agent's context window.
- $CL=0$: Only the current population is visible.
- $CL=1$ (Default): Current and one prior turn.
- $CL=20$ (Full): The entire session history is visible, including a specific prompt asking the agent to justify novelty relative to the archive.
Exploration (Selection Noise, $\epsilon$ ): An $\epsilon$ -greedy strategy where, with probability $\epsilon$ , the agent's selection is replaced by a random action (random parent selection, random mutation settings, or toggling color mode).
Multi-Agent Diversity ( $N_A$ ): The number of distinct "personalities" assigned to agents. Personality traits (e.g., "You are searching for the visual equivalent of a wailing siren") were generated by an LLM and prepended to the system prompt to simulate diverse playstyles.

Evaluation Metrics

The authors developed metrics to quantify the "x-factor" of the human archive:

Semantic Recall: The ability to rediscover known objects (based on the THINGS dataset) by measuring the minimum cosine distance between archive images and class embeddings in a joint text-vision space (SigLIP2-B).
Visual & Semantic Novelty: Measured via $k$ -covering radius in embedding spaces (SigLIP-2-B-alignet for images, gemini-embedding-001 for captions). This quantifies how well the archive covers the space of possible forms.
Phylogenetic Tree Balance ( $J_1$ ): A measure of the diversity of the evolutionary lineage, assessing whether agents branch from a wide variety of ancestors or repeatedly from the same few images.

Key Results

1. Exploration vs. Quality Trade-off

Noise ( $\epsilon$ ): Introducing small amounts of exploratory noise ( $\epsilon \approx 0.25$ ) increased the diversity of the archive (Visual/Semantic Coverage) and prevented mode collapse. However, higher noise levels ( $\epsilon \ge 0.5$ ) degraded the legibility and semantic recall of the images.
Observation: While noise increased phylogenetic diversity (agents branched from more varied sources), it did not necessarily increase genetic diversity in a way that led to deeper, meaningful long-term variation. VLMs exhibited a "stubborn favoritism," consistently selecting specific forms even from noisy sets.

2. The Role of Memory (Context Length)

$CL=0$: Agents suffered from severe mode collapse, publishing duplicates and near-identical variations, resulting in low Semantic Recall.
$CL=1$: Surprisingly effective. A single step of context was sufficient to prevent immediate repetition and yielded the highest Semantic Recall scores.
$CL > 1$: Increasing context length generally degraded performance. $CL=2$ and $CL=10$ led to "pathological behavior," including information overload, messier images, and auto-sycophantic loops where agents reinforced their own biases.
$CL=20$: While diversity metrics peaked (likely due to the explicit novelty prompt), Semantic Recall remained lower than at $CL=1$, and images were often abstract or noisy.

3. Multi-Agent Dynamics

Diversity: Increasing the number of agents ( $N_A$ ) significantly improved Visual and Semantic Coverage and Tree Balance.
Adversarial Artifacts: At high agent counts ( $N_A = 1,000$ ), the archive became rife with high-frequency, uninterpretable, "psychedelic" patterns. These images acted as adversarial hubs, satisfying the conflicting, abstract preferences of many different agents simultaneously. While these images achieved high Semantic Coverage scores, they were qualitatively non-descript and noisy.

4. Comparison to Human Baseline

The historical human archive dominated most metrics, particularly in Semantic Recall and the refinement of images. Humans demonstrated a capacity for "bold leaps" in semantic space (e.g., jumping from a face to a car), whereas VLMs tended to make smaller, incremental steps (e.g., face $\to$ eye $\to$ frog $\to$ car).

Significance and Claims

The paper claims that fully automated open-endedness has not yet been achieved by VLMs, even with state-of-the-art models. However, the study provides a framework for understanding the necessary conditions:

Minimal Memory is Optimal: Contrary to the intuition that more history aids reasoning, the authors found that excessive context leads to pathological behavior and mode collapse. A minimal context ($CL=1$) was sufficient for optimal performance.
Exploration is a Double-Edged Sword: While necessary to escape local optima, too much noise degrades the quality of the artifacts.
Diversity without Cohesion: Multi-agent systems increase coverage of the search space but can lead to the propagation of "adversarial" images that satisfy many agents but lack semantic meaning.
The "X-Factor" Gap: The authors conclude that VLMs currently lack the "boldness and discernment" of human users. While they can be guided to produce diverse artifacts, they struggle to generate the "stepping stones" of high-quality, semantically coherent novelty that characterize human-driven open-ended search.

The work suggests that to achieve human-level open-endedness, future systems may need to combine these interventions carefully and potentially incorporate user studies to calibrate metrics, rather than relying solely on automated optimization. The code is made available to facilitate further research into the "ingredients" of open-endedness.