Scalable Question-Centric Text-to-Image Evaluation: Reliable Ranking, Fine-Grained Diagnosis, and Cost-Aware Routing

In the rapidly evolving world of artificial intelligence, a specific class of software has emerged that can conjure pictures from nothing but a sentence. These systems, known as text-to-image models, listen to a user's description—perhaps "a red cat sitting on a blue mat"—and generate a visual representation of that scene. For years, the primary way to judge how well these machines worked was to look at the final picture and assign it a single score. If the image looked good overall, the model was considered strong. However, as these tools have improved, a problem has arisen: different models often achieve similar overall scores while possessing very different strengths and weaknesses. One might be excellent at drawing animals but terrible at rendering text, while another might follow complex instructions perfectly but struggle with simple shapes. This makes it difficult for users to choose the right tool for a specific job, as a single number cannot reveal the intricate details of what a model can actually do.

To solve this, a team of researchers from China Unicom has introduced a new way of testing these systems called QC-T2I-Bench. Instead of treating a prompt as a single, indivisible block, their method breaks every request down into a series of tiny, specific questions. If a user asks for a red cat on a blue mat, the system does not just ask, "Is the image good?" It asks, "Is there a cat?" "Is the cat red?" "Is there a mat?" and "Is the mat blue?" Crucially, the researchers also map out how these questions depend on one another. They recognize that if the system fails to draw the cat, it cannot possibly succeed in drawing the cat's red fur. By organizing these questions into a logical structure, the team can pinpoint exactly where a model fails and whether that failure is a basic mistake or a more complex error that only happens when multiple requirements are combined.

The researchers tested this approach on thirteen different open-source image generators, using both English and Chinese prompts. They found that while some models performed well on simple tasks, their ability to handle complex requests dropped off sharply as the number of requirements increased. When a prompt contained requirements for just two different capabilities, the models succeeded in creating a perfect image about 80.7% of the time. However, when the request demanded seven or more distinct capabilities simultaneously, the success rate plummeted to 37.2%. This decline was not just a matter of adding more errors; it revealed that the models struggled significantly to hold all the pieces of a complex puzzle together at once. The study also showed that by understanding these dependencies, the researchers could distinguish between a model that simply cannot draw a specific object and one that can draw the object but fails when asked to add specific details to it.

Beyond simply ranking the models, this detailed breakdown allows for a more practical application: cost-aware routing. In the real world, running these image generators on powerful computers costs money and energy. The researchers demonstrated that the same detailed records used to diagnose errors could be used to automatically select the most efficient model for a given task without needing to train a new system. By analyzing the specific requirements of a user's request, their system could choose a generator that was cheaper to run but still met the quality standards. In their tests, this method matched the performance of a top-tier model while using 21.3% less computing power per image. This suggests that the future of using these tools lies not just in making them smarter, but in using a deeper understanding of their specific capabilities to match the right tool to the right job, saving resources while maintaining high quality.