💻 computer science

GEMS: Agent-Native Multimodal Generation with Memory and Skills

GEMS is an agent-native multimodal generation framework that integrates a structured multi-agent loop, persistent hierarchical memory, and on-demand domain skills to significantly enhance generation quality and enable lightweight models to outperform state-of-the-art baselines on complex and specialized tasks.

Original authors: Zefeng He, Siyuan Huang, Xiaoye Qu, Yafu Li, Tong Zhu, Yu Cheng, Yang Yang

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zefeng He, Siyuan Huang, Xiaoye Qu, Yafu Li, Tong Zhu, Yu Cheng, Yang Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very talented, but slightly naive, artist to paint a complex scene. You give them a prompt like, "Draw a futuristic city where the buildings are made of glass, floating in the sky, with a cat wearing a spacesuit walking on a rainbow."

If you ask a standard AI model (the "naive artist"), it might draw a city, or a cat, or a rainbow, but it often misses the specific details or gets the physics wrong. It tries once, gets it 80% right, and stops.

GEMS is like hiring a super-organized art director to manage that artist. Instead of just giving the prompt and walking away, GEMS sets up a workshop where the artist, the director, and a team of specialists work together until the painting is perfect.

Here is how GEMS works, broken down into three simple parts:

1. The Agent Loop: The "Try, Check, Fix" Cycle

Think of this as a rehearsal process.

The Artist (Generator): Makes a first draft of the image.
The Critic (Verifier): Looks at the draft and checks a checklist. "Is the cat wearing a suit? Yes. Is the cat walking on the rainbow? No, the cat is floating."
The Editor (Refiner): Takes the Critic's notes and tells the Artist exactly what to change for the next round.
The Result: Instead of giving up after one try, the system keeps looping through this process. It's like a musician practicing a song over and over, fixing one wrong note at a time, until the performance is flawless.

2. Agent Memory: The "Smart Notebook"

In previous systems, if the artist made a mistake in round 1, they might forget it by round 3, or the system would get confused by too much information.

GEMS uses a specialized notebook that remembers everything in two ways:

The Raw Facts: It keeps the actual images and the specific checklist items (e.g., "Round 1: Cat was floating").
The Wisdom: It doesn't just copy-paste the artist's internal thoughts (which can be long and messy). Instead, a "Summarizer" reads the thoughts and writes a short, punchy lesson: "Lesson learned: Cats need to be on the ground, not floating, unless specified."

This allows the system to look back at the entire history of the project without getting overwhelmed by noise, ensuring it doesn't make the same mistake twice.

3. Agent Skill: The "Toolbox of Experts"

Sometimes, a general artist just doesn't know how to do something specific, like drawing realistic text or complex spatial arrangements.

GEMS has a toolbox of specialized experts (Skills) that it can pull out only when needed:

Need a picture with perfect text? It pulls out the "Text Rendering Expert."
Need a picture with complex 3D depth? It pulls out the "Spatial Intelligence Expert."
Need a beautiful, artistic style? It pulls out the "Aesthetic Drawing Expert."

Crucially, these experts aren't always "on," which saves energy. They are only called in when the specific job requires them, making the system efficient and incredibly good at niche tasks.

The Big Win: Small Models, Big Results

The most exciting part of this paper is the result. Usually, to get amazing results, you need a massive, expensive, super-computer-sized AI model (like the "Nano Banana" mentioned in the paper).

GEMS proved that you don't need the biggest engine to win the race. By using this smart "Art Director" system (GEMS), they took a tiny, lightweight model (Z-Image-Turbo) and made it perform better than the massive, expensive models on difficult tasks.

In short: GEMS doesn't just make the artist smarter; it gives them a better process, a better memory, and the right tools for the job. It turns a "good enough" attempt into a masterpiece through smart, iterative teamwork.

1. Problem Statement

Recent multimodal generation models (e.g., GPT-Image, Nano Banana, Qwen-Image) have achieved state-of-the-art (SOTA) performance on general-purpose, straightforward text-to-image tasks. However, they face significant limitations in two key areas:

Complex Instructions: They struggle with intricate, multi-faceted prompts requiring logical reasoning or strict adherence to multiple constraints.
Specialized Downstream Tasks: They often fail in domain-specific applications (e.g., creative drawing, academic illustration, precise text rendering) where general-purpose capabilities reach their "long-tail" limits.

Existing solutions, such as inference-time scaling via simple prompt rewriting or basic iterative loops, suffer from insufficient guidance, excessive information redundancy, or a lack of adaptability to diverse downstream domains. Furthermore, specialized multi-agent frameworks are often fragmented and difficult to integrate into mainstream generative pipelines.

2. Methodology: The GEMS Framework

GEMS (Agent-Native Multimodal GEneration with Memory and Skills) reframes text-to-image generation as an iterative optimization problem driven by an agent-centric architecture. It consists of three core pillars:

A. Agent Loop (The Optimization Engine)

The Agent Loop serves as the backbone, orchestrating a closed-loop refinement process involving five collaborative modules:

Planner: Acts as the strategic entry point. It interacts with the Skill Manager to retrieve relevant domain expertise and synthesizes an enhanced initial prompt. It also dispatches the original prompt to the Decomposer.
Decomposer: Breaks down the user prompt into a set of atomic, binary (yes/no) visual criteria ( $C$ ) to enable fine-grained evaluation.
Generator: A model-agnostic module that synthesizes the visual output ( $I_i$ ) based on the current prompt ( $P_i$ ).
Verifier: Powered by a Multimodal Large Language Model (MLLM), it assesses the generated image against the atomic criteria, producing a binary feedback vector ( $V_i$ ).
Refiner: The core optimization engine. It analyzes the current state, historical context, and verification feedback to generate a refined prompt ( $P_{i+1}$ ) for the next iteration. The loop terminates when all criteria are met or a maximum iteration limit is reached, returning the best-performing image found.

B. Agent Memory (Persistent Trajectory Management)

Unlike previous systems that only retain the immediate previous step or the best state, GEMS maintains a persistent, trajectory-level memory.

Hierarchical Compression: To balance information density and token efficiency, the system categorizes history into two tiers:
- Factual Artifacts: Prompts, images, and verification feedback are stored in raw form to ensure objective accuracy.
- Experiential Summaries: Verbose reasoning traces (Chain-of-Thought) are processed by a Compressor to distill them into concise, high-level "Experiences" ( $E_i$ ).
This hybrid memory state ( $M_i$ ) allows the Refiner to access a global view of the optimization process, eliminating redundancy while preserving strategic insights.

C. Agent Skill (Domain Expertise)

To address the fragmentation of specialized tasks, GEMS introduces an extensible Agent Skill repository.

On-Demand Loading: The system uses a lightweight manifest (names and descriptions) that is always loaded. Detailed domain-specific instructions are fetched only when a specific skill is triggered.
Progressive Exposure: This mechanism allows the system to scale to a vast library of expertise (e.g., Creative Drawing, Spatial Intelligence, Text Rendering) without imposing a cognitive or computational burden on the reasoning process.
Integration: The Planner matches user intent with these skills to inject domain-specific constraints into the prompt before the iterative loop begins.

3. Key Contributions

GEMS Framework: A novel agent-native framework that unifies iterative refinement, persistent memory, and domain skills to overcome the limitations of foundational models in complex generation tasks.
Hierarchical Memory Mechanism: A persistent memory system that utilizes hierarchical compression to manage historical context, effectively balancing factual accuracy with strategic reasoning.
Extensible Skill Module: A scalable architecture for domain-specific expertise that enables on-demand loading, allowing the system to handle diverse downstream applications without architectural fragmentation.
Empirical Validation: Extensive experiments demonstrating that agent-based reasoning can unlock the potential of smaller, lightweight models, enabling them to outperform much larger SOTA models.

4. Experimental Results

The authors evaluated GEMS across five mainstream benchmarks (GenEval, GenEval2, DPG-Bench, OneIG, WISE) and four specialized downstream tasks (LongText-Bench, SpatialGenEval, CREA, ArtiMuse) using two backends: Z-Image-Turbo (6B parameters) and Qwen-Image-2512 (20B parameters).

Mainstream Performance:
- Using the lightweight Z-Image-Turbo, GEMS achieved an average performance gain of 14.22 points across mainstream benchmarks.
- Crucial Finding: GEMS enabled the 6B Z-Image-Turbo to surpass the SOTA closed-source model Nano Banana 2 on the challenging GenEval2 benchmark (Score: 63.5 vs. 44.6).
Downstream Performance:
- GEMS achieved an average improvement of 14.03 points on downstream tasks with Z-Image-Turbo, significantly outperforming the best inference-time scaling baseline by +8.92 points.
- On Qwen-Image-2512, GEMS improved performance by 16.24 (mainstream) and 7.96 (downstream) points.
Ablation Studies:
- Agent Loop: Provided the largest boost (e.g., +21.4 points on GenEval2), proving the value of iterative refinement over single-shot generation.
- Agent Memory: Adding hierarchical memory improved scores by +9.0 points, confirming that compressed experiences are more effective than raw reasoning logs.
- Agent Skill: Contributed an additional +2.1 points, showing that domain-specific guidance is critical for specialized tasks.
Efficiency: Despite the iterative nature, GEMS achieved superior performance with fewer average generated images (approx. 2.8 iterations) compared to other scaling methods, due to its early stopping mechanism and efficient memory usage.

5. Significance

The paper demonstrates that agentic reasoning and domain-specific expertise can effectively extend the capabilities of foundational models beyond their inherent parameter limits.

Democratization of SOTA: It proves that lightweight, distilled models (6B) can outperform massive closed-source models when equipped with a sophisticated agent framework.
Paradigm Shift: It moves multimodal generation from a "single-shot" paradigm to a "collaborative optimization" paradigm, addressing the "long-tail" challenges of complex instructions and specialized applications.
Scalability: The modular design (Loop, Memory, Skill) offers a scalable blueprint for future multimodal agents, suggesting that the integration of memory and skills is as critical as model size for achieving high-fidelity generation.