💻 computer science

DyCoRM: Dynamic Criterion-Aware Reward Modeling for Text-to-Image Generation

This paper introduces DyCoRM, a dynamic criterion-aware reward modeling framework that addresses the need for fine-grained, task-specific image evaluation in text-to-image generation by leveraging a newly constructed dataset and benchmark to enable more precise alignment with user requirements.

Original authors: Jiaying Qian, Ziheng Jia, Qian Zhang, Zicheng Zhang, Jiayi Guo, Junqi Zhang, Guangtao Zhai, Xiongkuo Min

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiaying Qian, Ziheng Jia, Qian Zhang, Zicheng Zhang, Jiayi Guo, Junqi Zhang, Guangtao Zhai, Xiongkuo Min

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an art critic, but instead of just saying "I like this painting more than that one," you have to explain exactly why based on a specific rule the artist gave you.

This paper introduces a new system called DyCoRM (Dynamic Criterion-Aware Reward Model) designed to help computers judge AI-generated images. Here is the breakdown using simple analogies:

The Problem: The "One-Size-Fits-All" Judge

Currently, most AI image judges work like a general manager who only looks at the final product and gives a single score (e.g., "8 out of 10").

The Issue: Sometimes a user wants an image that is "scary," and other times they want one that is "colorful." A general manager might give a high score to a colorful image even if the user specifically asked for something scary. The old judges don't know how to switch their focus based on the specific request. They are stuck using a fixed set of rules for every single job.

The Solution: The "Specialized Inspector" (DyCoRM)

The authors built a new system that acts like a flexible, specialized inspector. Instead of just giving a score, this inspector does two things in order:

Step 1: Read the Blueprint (Criterion Grounding):
Before looking at the images, the inspector reads the user's prompt and asks, "What are the specific rules for this job?"
- Analogy: If the prompt is "a cyberpunk city," the inspector writes down: "Check for neon lights, flying cars, and dark streets."
- If the prompt is "a cozy kitchen," the inspector writes down: "Check for warm lighting, steam on the window, and realistic food textures."
- The system learns to figure out these specific rules automatically for every new request.
Step 2: Grade Based on the Rules (Criterion-Conditioned Comparison):
Once the rules are set, the inspector looks at two images and compares them only based on those specific rules.
- Analogy: It doesn't just say "Image A is prettier." It says, "Image A wins because the neon lights are brighter (Rule #1), but Image B wins because the food looks tastier (Rule #2)."

The Training Data: The "Practice Exam" (DyCoDataset-20K)

To teach this inspector how to do its job, the researchers created a massive new training set called DyCoDataset-20K.

How they made it: They took thousands of prompts, generated images from 14 different AI models, and then hired humans to act as "tutors."
The Tutoring Process: The humans didn't just pick a winner. They had to:
1. Write down the specific criteria for that specific prompt (e.g., "The hands must look realistic").
2. Compare the images based only on those criteria.
The Result: A dataset of 20,000+ examples where the "rules" change for every single task, teaching the AI to be flexible.

The Benchmark: The "Final Exam" (DyCoBench-1K)

They also created a smaller, harder test set called DyCoBench-1K. This is like a final exam where the questions are tricky and require the AI to switch its focus rapidly. The results showed that DyCoRM passed this exam much better than previous "general manager" style judges.

The Application: The "Personal Shopper" (DyCoPick)

Finally, the authors showed how to use this system in real life with a tool called DyCoPick.

How it works: Imagine you ask an AI to generate 10 pictures of a "cat in space."
The Old Way: The AI might just pick the first one it thinks looks "good" generally.
The DyCoPick Way: The system generates 10 options, then uses the "Specialized Inspector" to look at them based on your specific needs (e.g., "I want the cat to look fluffy" or "I want the background to be dark"). It then picks the one image that best fits your specific criteria, acting like a personal shopper who knows exactly what you want, not just what is popular.

Summary

In short, this paper says: "Stop using one generic rulebook for all AI image judging. Instead, build a system that first figures out what the specific rules are for the current task, and then judges the images based on those specific rules." They built the teacher (the dataset), the student (the model), and the test (the benchmark) to prove this works better than the old way.

Technical Summary: DyCoRM: Dynamic Criterion-Aware Reward Modeling for Text-to-Image Generation

1. Problem Statement

While text-to-image (T2I) generation models have advanced significantly in producing high-quality images, user demands are shifting toward outputs that satisfy specific, task-relevant requirements. Existing reward models and evaluation frameworks largely rely on static, predefined dimensions or aggregate preference signals. These approaches fail to adapt to the dynamic nature of user judgment, where evaluation criteria vary significantly based on the prompt, the image pair, and the intended use case (e.g., prioritizing text readability for design tasks versus emotional tone for storytelling).

The core challenge identified is that current models cannot decompose evaluation into two distinct steps: (1) inferring the task-relevant criteria from the prompt-image context, and (2) modeling image preferences conditioned on those specific criteria. Consequently, existing models struggle to prioritize the correct aspects of an image when different tasks require different evaluation standards.

2. Methodology

2.1 Data Construction: DyCoDataset-20K and DyCoBench-1K

To enable dynamic, criterion-aware modeling, the authors constructed a new dataset and benchmark:

DyCoDataset-20K: A dataset comprising 4,876 prompts, 23,378 image pairs, and 94,572 fine-grained criteria. The construction pipeline involves:
- Prompt Generation: Using GPT-4o to generate diverse prompts controlled by six components (subject, appearance, scene, motion, format, rendering) to vary complexity.
- Image Acquisition: Generating 5–6 images per prompt from a pool of 14 models (including proprietary and open-source) to ensure diverse comparisons.
- Human Annotation: A two-stage protocol where annotators first formulate fine-grained criteria relevant to the specific prompt-image pair, and then perform pairwise comparisons under each finalized criterion. Labels are retained only if >70% of annotators agree.
DyCoBench-1K: A curated benchmark derived from the dataset, containing 192 prompts and 822 image pairs. It is designed to test dynamic evaluation capabilities, including preference reversal cases where the winner changes based on the selected criterion.

2.2 Model Architecture: DyCoRM

DyCoRM is a two-stage reward modeling framework:

Stage 1: Criterion Grounding: The model takes a prompt and an image pair as input and generates a set of task-relevant fine-grained criteria ( $C = \{c_1, c_2, ..., c_M\}$ ). This is trained as an autoregressive sequence generation task using cross-entropy loss against human-annotated criteria.
Stage 2: Criterion-Conditioned Reward Learning: Given the prompt, image pair, and a specific criterion (or set of criteria), the model predicts the preference between the images. The authors investigated both pointwise and pairwise loss functions, finding that pairwise loss consistently yields better performance. The model learns to output a probability indicating which image is preferred under the specified condition.

2.3 Application: DyCoPick

DyCoPick extends the reward modeling framework to the generation selection phase. It operates in three stages:

Candidate Generation: Multiple T2I models generate candidates for a given prompt.
Criteria Resolution: If the user provides criteria, they are used; otherwise, the system automatically derives task-relevant criteria from the prompt and candidates.
Criterion-wise Selection: DyCoRM performs pairwise comparisons among candidates under each criterion to select the best output for personalized preference alignment.

3. Key Contributions

DyCoRM Framework: The first reward modeling framework that explicitly decomposes evaluation into criterion grounding and criterion-conditioned preference learning, allowing models to adapt judgments to diverse task requirements.
DyCoDataset-20K & DyCoBench-1K: The construction of a large-scale dataset with dynamic, fine-grained criteria and a benchmark specifically designed to evaluate reward models under dynamic, criterion-specific judgments, moving beyond static overall preference labels.
DyCoPick: A practical application demonstrating how dynamic, criterion-aware reward modeling can be integrated into T2I generation pipelines for personalized output selection.

4. Experimental Results

The authors evaluated DyCoRM on DyCoBench-1K and four cross-domain benchmarks (ImageReward, Pick-a-Pic, HPDv2, HPDv3).

Performance on DyCoBench-1K: DyCoRM significantly outperformed both specialized T2I reward models (e.g., HPSv3, ImageReward) and general Large Multimodal Models (LMMs) across single-criterion, multi-criteria, and overall preference settings. For instance, in the overall preference setting, DyCoRM achieved an accuracy of 0.791 (Cohen's Kappa 0.772), compared to 0.732 for the next best baseline (HPSv3).
Generalization: On cross-domain benchmarks, DyCoRM maintained competitive performance, demonstrating that its training on dynamic criteria generalizes to standard overall-preference evaluation tasks.
Dynamic Capability: Analysis showed that DyCoRM's performance remains robust across varying task difficulties and criterion counts, indicating it adapts to changing evaluation conditions rather than relying on fixed heuristics.
Human Study (DyCoPick): In a human study comparing DyCoPick against other reward models as selectors, DyCoPick was preferred across all settings (overall, single-criterion, and multi-criteria), with particularly strong advantages in criterion-specific evaluations.

5. Significance and Claims

The paper claims to establish the first reward modeling framework for dynamic and fine-grained evaluation in T2I generation. By shifting the paradigm from static preference prediction to dynamic, criterion-aware evaluation, the work addresses the limitation of current models that cannot adapt to specific user requirements. The authors posit that this approach enables more precise alignment between generated outputs and user intent, bridging the gap between offline evaluation and practical generation-time selection. The study emphasizes that while dynamic criterion collection is more expensive than static preference labeling, the resulting dataset and framework provide a necessary foundation for the next generation of personalized T2I systems.