🤖 AI

Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence

This paper introduces the "Agentic Self-Improvement" framework, a two-stage closed-loop optimization system that combines multimodal LLM-driven prompt refinement with Bayesian optimization to significantly enhance the reliability, adherence, and quality of black-box Image-to-Video generation, outperforming traditional trial-and-error methods in human preference studies.

Original authors: Aman Tyagi, Hemanth Boinpally, Jonathan Chen, Douglas Gebert, Steven Hickson

Published 2026-08-13

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Aman Tyagi, Hemanth Boinpally, Jonathan Chen, Douglas Gebert, Steven Hickson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a magical, invisible artist to paint a picture based on a description you give it. You say, "Draw a cat sitting on a red rug," and the artist paints a masterpiece. But then you ask for a second picture with the exact same words, and suddenly, the cat is blue, the rug is gone, and the cat is floating in space. This is the current reality of "Image-to-Video" AI. These are powerful computer programs that can turn a single photo and a sentence into a moving video. They are like dream machines, capable of creating anything you can imagine. However, they are also incredibly unpredictable. They work on a bit of luck, meaning that if you change a tiny setting or even just wait a second longer before hitting "generate," the result might be completely different. For professional creators who need to make a specific scene for a movie or an ad, this "roll the dice" approach is a nightmare. They can't just hope for the best; they need to know the AI will do exactly what they ask, every single time.

This paper introduces a new way to tame these wild digital artists. Instead of just asking the AI to "try again" over and over until something good happens (a process called trial-and-error), the authors built a smart, self-correcting system they call "Agentic Self-Improvement." Think of it as giving the AI a very strict, super-smart editor and a GPS. The system doesn't just guess; it plans. It breaks your request down into a checklist of specific questions, checks the video it made against that checklist, and then rewrites your request or tweaks the settings to fix any mistakes. It does this in a loop, getting better and better until the video matches your vision perfectly. The result is a method that turns video creation from a game of chance into a reliable, step-by-step process, making it much easier to get exactly the video you want without wasting hours of computer time.

The Problem: The "Roll the Dice" Video Maker

Imagine you are a director trying to film a scene where three women are sitting on the steps of a building. You have a photo of the building and you type a prompt: "A group of women sitting on the steps of a building." You hit the button, and the AI generates a video. But when you watch it, something is wrong. Maybe one woman has six fingers, or the building's wall changes color in the middle of the clip, or the women disappear and reappear like ghosts.

This happens because current AI video models are "black boxes." They are powerful, but they are also "stochastic," which is a fancy word for "random." Even if you type the exact same words and use the same photo, the AI might give you a totally different video every time because of tiny, invisible settings called "seeds" and "guidance scales." To get the perfect video, creators currently have to use a brute-force method: they generate dozens or hundreds of videos, hoping one of them will be right. It's like trying to find a specific key in a giant pile of keys by just grabbing them one by one. It takes forever, costs a lot of money in computer power, and is incredibly frustrating.

The Solution: The "Agentic" Detective

The authors of this paper propose a smarter way. Instead of blindly generating videos, they created a system that acts like a detective or a coach. They call this the "Agentic Self-Improvement" framework. It works in two main stages, like a two-step dance to perfect the video.

Stage 1: The Prompt Optimizer (The Editor)
First, the system looks at your original request and your photo. It uses a super-smart AI (called a multimodal Large Language Model, or mLLM) to act as an editor. This editor doesn't just read your words; it breaks them down into a specific checklist of questions.

The "DSG" Questions: These are like a logic puzzle. If you said "women sitting on steps," the AI asks: "Are the women present?" "Are they sitting?" "Are they on steps?" "Is there a building?"
The "CMQ" Questions: These catch the weird mistakes AI often makes. It asks: "Do the women's faces stay the same?" "Are their hands realistic?" "Do they disappear and reappear?"

The system generates a video, then the "editor" AI watches it and answers these questions with "Yes" or "No." If the AI says "No" to "Are the women sitting?", the system knows something is wrong. It then rewrites your prompt to be clearer, like changing "women sitting" to "three women sitting firmly on the steps." It repeats this process, generating and checking, until the video passes all the questions.

Stage 2: The Hyperparameter Optimizer (The Tuner)
Once the prompt is perfect, the system still needs to find the right "knobs" to turn on the AI machine. These knobs are the random "seed" (which decides the specific random details) and the "CFG scale" (which decides how strictly the AI follows your instructions).
Instead of guessing these numbers, the system uses a math technique called Bayesian Optimization. Imagine you are trying to find the hottest spot on a beach. A random search would just run around blindly. Bayesian optimization is like having a map that learns from every step you take. If the water is cold at point A, the map tells you to look closer to point B. The system uses this to efficiently find the perfect combination of seed and settings that produces the best video, without wasting time on bad combinations.

How They Tested It: The Human Vote

To see if this actually works, the researchers ran a big test. They took 100 different photo-and-prompt pairs and asked their "Agentic" system to make videos. They compared these videos against videos made by the old "random search" method.

They didn't just rely on computers to judge the quality; they asked real humans to watch the videos and pick the winner. The results were clear:

When the system was allowed to try 100 times (a "100-generation budget"), the videos made by their smart system were preferred by humans 69% of the time over the random videos.
Even with a smaller budget of just 10 tries, their system still won 47% of the time, compared to only 14% for the random method.

The paper also checked if the "editor" AI was actually good at spotting mistakes. They compared the AI's answers to human answers on 100 video clips. The AI was 92% accurate on the logic questions (like "is the woman present?") and 82% accurate on the tricky motion questions. This high level of agreement proved that the AI could reliably act as a judge.

What This Means

The paper suggests that by treating video creation as a goal-directed optimization problem—where the AI constantly checks its work and fixes itself—we can move beyond the current "speculative" phase of video generation. Instead of hoping for a lucky result, creators can use this framework to get reliable, high-quality videos that stick to their original vision.

The authors are careful to note that this isn't a magic wand that solves every problem instantly. The system still takes time and computer power to run the loops. Also, the "editor" AI is still limited by how well current computers understand complex motion and time. However, the study shows that this "Agentic" approach is a significant step forward. It turns the chaotic, trial-and-error process of making AI videos into a structured, controllable workflow, making these powerful tools much more useful for real-world jobs like filmmaking and advertising.

Technical Summary: Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence

1. Problem Statement

Modern black-box Image-to-Video (I2V) models possess significant generative capabilities but suffer from a lack of fine-grained control and reliability, creating a barrier for professional workflows. The core issue is the inherent stochasticity and sensitivity of these models: minor variations in textual prompts or hyperparameters (such as random seeds or Classifier-Free Guidance (CFG) scales) often yield drastically different and unpredictable outputs. Consequently, practitioners are forced into inefficient, brute-force "trial-and-error" workflows, generating numerous video variations in hopes of one aligning with their creative intent. This paper argues that bridging the gap between generative potential and production-quality requirements necessitates moving beyond speculative generation toward a systematic, closed-loop optimization framework.

2. Methodology: Agentic Self-Improvement

The authors propose the "Agentic Self-Improvement" framework, which reframes video synthesis as a goal-directed, closed-loop optimization problem. The system operates as a model-agnostic wrapper that systematically navigates the generation parameter space through a novel two-stage approach:

Stage 1: Prompt Optimization via Structured Evaluation

This stage addresses the semantic gap between user intent and model output by iteratively refining the input prompt using a multimodal Large Language Model (mLLM).

Automated Question Generation: The mLLM generates two types of evaluative queries based on the input prompt and image:
- Davidsonian Scene Graph (DSG) Questions: These deconstruct the prompt into verifiable semantic queries regarding agents, actions, states, and object locations (e.g., "Are the women sitting on steps?").
- Common Mistake Questions (CMQ): These target known failure modes of video generation, such as temporal inconsistency, anatomical errors, or visual artifacts (e.g., "Are facial features stable and free from distortion?").
Iterative Refinement Loop: The I2V model generates a video, which is evaluated by the mLLM acting as a Visual Question Answering (VQA) system. The mLLM provides binary (Yes/No) answers to the DSG and CMQ questions. Based on the "No" answers, the mLLM rewrites the prompt to address specific shortcomings. This loop repeats for a fixed number of iterations to maximize the alignment score.
Reliability Validation: The authors validated the mLLM (Gemini 2.5 Pro) as an automated evaluator against human ground truth, achieving 92% accuracy on DSG questions and 82% on CMQ questions.

Stage 2: Hyperparameter Optimization via Bayesian Search

Once an optimal prompt is identified, the framework optimizes the model's hyperparameters to further reduce variance.

Co-Optimization: The system uses Bayesian optimization (specifically the Vizier platform) to simultaneously search for optimal stochastic seeds and CFG scales.
Handling Stochasticity: To address the non-smooth nature of random seeds (where numerical proximity does not imply similar outputs), the seed is treated as an unordered categorical variable within a Gaussian Process Bandit algorithm.
Reward Function: The search is guided by a multi-objective reward function comprising:
- Perceptual Quality: Measured by RAHF (Rich Automated Human Feedback) and UVQ (Universal Video Quality).
- Temporal Coherence: Evaluated via shotcut detection, flow continuity, and loop detection.
- Video-Text Adherence (VTA): A novel metric derived directly from the DSG and CMQ evaluations. VTA aggregates binary responses into a weighted score, recursively masking child node scores if parent nodes fail (e.g., if "Is there a dog?" is false, "Is the dog running?" is zeroed).

3. Key Contributions

The paper outlines four primary contributions:

Agentic Self-Improvement Framework: A systematic, goal-directed control mechanism for black-box I2V models that replaces open-loop speculation with closed-loop optimization.
Structured Prompt Optimization: A method utilizing an mLLM with automatically generated DSG and CMQ evaluations to iteratively refine prompts for semantic and artifact-free adherence.
Bayesian Hyperparameter Co-Optimization: An efficient search strategy that jointly optimizes stochastic seeds and CFG scales, outperforming blind random sampling.
Video-Text Adherence (VTA) Metric: A novel, structured metric derived from DSG and CMQ evaluations to quantitatively guide optimization and score final content.

4. Experimental Results

The framework was evaluated using the Veo 2.0 model on 100 image-prompt pairs from V-Bench, comparing the agentic approach against unguided random search baselines.

Prompt Optimization Impact: In a head-to-head comparison, videos generated using the optimized prompts were preferred by human evaluators 27% of the time, compared to only 5% for the baseline (with 68% ties), indicating a significant perceptual improvement.
End-to-End Performance:
- Against a random search baseline with a 100-generation budget, the agentic framework achieved human preference win rates ranging from 60% to 69%, depending on the selection metric used.
- The RAHF (Rich Automated Human Feedback) metric yielded the highest win rate at 69%.
- Even when compared against a stronger "Best-of-Random" baseline (which selects the best video from 100 random trials using the same automated metrics), the Bayesian optimizer decisively outperformed the baseline (e.g., 42% vs. 8% win rate for RAHF), confirming that the search algorithm itself guides the generation toward higher-quality regions rather than relying solely on post-hoc sorting.
Statistical Significance: The win rates were statistically significant ( $p < 10^{-6}$ for the 100-run budget), with 95% Wilson score confidence intervals ranging from 50.2% to 77.2% across different metrics.

5. Significance and Claims

The paper claims that this work provides a practical and extensible methodology for enhancing the predictability and control of state-of-the-art video generation models. By treating the prompt as a refinable target and employing intelligent hyperparameter search, the framework transforms video generation from a speculative, trial-and-error process into a reliable, production-ready tool.

The authors emphasize that their approach leverages the "generator-critic" paradigm, where the mLLM acts as a sophisticated visual critic using structured rubrics (DSG/CMQ) to steer the generative model. While acknowledging current limitations in mLLM understanding of complex motion and the computational cost of the framework, the work positions itself as a critical step toward moving I2V systems beyond "speculative curiosities" toward robust engines capable of meeting the precise demands of professional content creation.