DreamOmni3: Scribble-based Editing and Generation
The paper introduces DreamOmni3, a unified model that advances GUI-based creation by enabling flexible scribble-based editing and generation through a novel data synthesis pipeline and a joint input framework that combines original and sketched images for precise visual control.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are talking to a super-smart artist who can draw anything you describe. If you say, "Draw a cat," they do it. If you say, "Make the cat wear a hat," they change the picture. This is the world of AI image generation and editing, a field where computers learn to create and modify pictures based on your words. For a long time, the only way to talk to these digital artists was through text. But here's the problem: words are sometimes clumsy. Trying to describe exactly where to put a new object, or how to change a specific part of a messy drawing, can be like trying to give directions to a friend using only a map with no landmarks. You might say, "Put the tree on the left," but the AI might put it on the far left, or the wrong left.
To fix this, scientists have been exploring a new way to talk to these artists: scribbling. Instead of just typing, you can grab a digital pen and draw circles, boxes, or messy doodles directly on the screen to show the AI exactly what you mean. It's like pointing at a spot on a map and saying, "Here!" rather than describing the street name. This paper, DreamOmni3, dives deep into making this "point-and-draw" method work perfectly. It tackles the tricky parts of teaching computers to understand not just your words, but also your messy, hand-drawn lines, and combines them to create or edit images with incredible precision.
The Problem: When Words Aren't Enough
Imagine you are playing a game where you have to edit a photo. You want to replace a toy in a red circle with a handbag. If you just type "put a handbag here," the AI might get confused. Which toy? Where is "here"? Is the circle red or black? Is it the first image or the second? Language is great for big ideas, but it's terrible at pointing to specific, tiny details on a screen.
The authors of this paper realized that while AI has gotten really good at following text instructions, it often misses the "where" and the "how" when things get complicated. Users need a way to be more direct. They need to be able to draw a circle around a face and say, "Change this hair," or draw a squiggly line and say, "Put a car here." This is the concept of scribble-based editing and generation. It's about letting users mix text, images, and their own freehand drawings to control the AI.
The Solution: DreamOmni3
The team behind DreamOmni3 decided to build a model that treats scribbles as a first-class citizen, right alongside text and images. But there was a huge hurdle: data. AI models are like students; they need to see thousands of examples to learn. There were no existing textbooks (datasets) that showed AI how to interpret a messy hand-drawn circle and turn it into a perfect edit.
So, the first thing the authors did was invent a data factory. They took existing images and used a clever pipeline to create millions of new training examples.
- For editing: They took photos, found objects, and then manually (or with helper AI) drew circles, boxes, and doodles over them. They created four types of tasks:
- Scribble + Text: "Put a car in the red circle."
- Scribble + Text + Image: "Put the car from this picture into the red circle."
- Image Fusion: Taking an object from one photo and pasting it into another, guided by a scribble.
- Doodle Editing: Turning a rough sketch into a realistic object in the photo.
- For generation: They did the same thing but started with a blank white canvas instead of a photo, teaching the AI to draw new things based on where you scribbled.
They generated a massive dataset with tens of thousands of these examples (32,000 for multimodal editing, 29,000 for multimodal generation, and more for the other tasks) to teach the model how to read the user's mind through their drawings.
The Magic Trick: How the Model "Sees" the Scribble
Here is where the paper gets really clever. Usually, when you want an AI to edit a specific part of an image, you use a binary mask. Think of a mask like a stencil: it's a black-and-white image where white means "change this" and black means "leave this alone."
The authors argue that masks are too rigid. If you have three different things to change in one picture, you need three different black-and-white masks. That's messy and hard to describe in words. Instead, DreamOmni3 uses colored scribbles. You can draw a red circle for the car, a blue square for the tree, and a green line for the sky. The AI can easily tell them apart by color.
But there's a catch: if you draw a red circle over a car, the AI can't see the car anymore because the red ink is covering it! To solve this, DreamOmni3 uses a joint input scheme.
- It feeds the AI two images at the same time: the original photo and the photo with the scribbles on top.
- It uses a special encoding system (a way of labeling pixels) that tells the AI, "Hey, the red circle in this image is at the exact same spot as the car in that image."
- This allows the AI to see the scribble (to know what you want) and the original pixels (to know what it's changing), ensuring the edit is precise and the rest of the picture stays perfect.
The Brain: Teaching the AI to Understand "Messy"
The authors also realized that human drawings are often messy. A circle might look like an egg; a line might be wobbly. To help the AI understand these imperfect drawings, they introduced a Visual Language Model (VLM)—a type of AI that is really good at reasoning.
They trained this VLM alongside the image generator. The VLM acts like a translator. It looks at your scribble and the text you wrote, figures out what you really meant (e.g., "Oh, they drew a wobbly circle, but they clearly want a car"), and then tells the image generator exactly what to do. This joint training ensures the AI doesn't just follow rules blindly but actually understands the intent behind the scribble.
The Results: Does It Work?
The team tested DreamOmni3 on a new benchmark they created, filled with real-world images and tricky tasks. They compared it against other top models, including some famous commercial ones like GPT-4o and Nano Banana.
The results were impressive. DreamOmni3 consistently outperformed other open-source models and matched or beat the commercial giants in many areas.
- Human Evaluators (real people) rated DreamOmni3's success rate at 55.88% for editing tasks, which was higher than GPT-4o (59.56% in one metric, but DreamOmni3 was more consistent in others) and significantly higher than others like Omnigen2 (2.94%).
- In generation tasks, DreamOmni3 achieved a 54.55% success rate, again beating most competitors.
- The paper notes that while commercial models are strong, they sometimes struggle with specific scribble instructions, often leaving the scribbles in the final image or getting the proportions wrong. DreamOmni3, however, was specifically designed to handle these nuances, producing cleaner results where the scribbles disappeared and the edits looked natural.
Why This Matters
DreamOmni3 isn't just about making AI smarter; it's about making AI more human. It bridges the gap between what we imagine and what we can describe. By letting us draw, point, and doodle, it turns image editing from a technical chore into a creative conversation. The paper suggests that this approach—combining text, images, and freehand drawing with smart data synthesis and joint training—is the future of how we will interact with digital creation tools. It shows that when we stop trying to force computers to understand our vague words and start letting them see our direct actions, the results are much more magical.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.