💻 computer science

GenExam: A Multidisciplinary Text-to-Image Exam

GenExam is the first multidisciplinary benchmark for text-to-image generation that evaluates models through 1,000 exam-style prompts across 10 subjects with ground-truth images and fine-grained scoring, revealing significant performance gaps between open-source and leading closed-source models while assessing their integrated understanding, reasoning, and generation capabilities.

Original authors: Zhaokai Wang, Penghao Yin, Xiangyu Zhao, Changyao Tian, Yu Qiao, Wenhai Wang, Jifeng Dai, Gen Luo

Published 2026-03-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhaokai Wang, Penghao Yin, Xiangyu Zhao, Changyao Tian, Yu Qiao, Wenhai Wang, Jifeng Dai, Gen Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've spent years teaching a robot to draw. You show it pictures of cats, cars, and sunsets, and it gets pretty good at making them look realistic. But then, you hand it a final exam from a university.

The exam doesn't ask for a pretty picture of a cat. It says: "Draw a specific map of Italy in 800 BC, color the Etruscan lands orange, the Gauls yellow, and make sure the borders are exactly right. Also, draw a chemical reaction where a molecule changes shape, but don't mess up the atoms."

This is exactly what the paper GenExam is about. It's a new, super-hard "test" designed to see if AI can actually think and reason like an expert, not just copy what it's seen before.

Here is the breakdown of this "exam" using some everyday analogies:

1. The Problem: The "Art Student" vs. The "Engineer"

Currently, most AI image generators are like talented art students. They are great at style. If you say "paint a sunset," they make a beautiful sunset. But if you say "draw a blueprint for a bridge that can hold 50 tons," they might draw a bridge that looks nice but collapses if you put a feather on it.

Existing tests for AI only check if the picture looks pretty or if the words match the picture. They don't check if the picture is logically correct.

Old Test: "Does this look like a graph?" (Yes, it has lines.)
GenExam: "Is the line actually the equation $y=x^2$ ? Does it pass through the exact point (1,1)? Is the shading in the right spot?"

2. The Solution: GenExam (The "Final Exam")

The researchers created GenExam, which is like a university final exam for AI.

The Subjects: It covers 10 different subjects, from Math and Physics to History and Music.
The Questions: The prompts are like real exam questions. They are precise, complex, and require deep knowledge.
- Example: "Draw a food web for the Arctic tundra. Make sure the lemming eats the lichen, but the bear doesn't eat the lichen directly."
The Answer Key: Every question comes with a "Ground Truth" (the perfect answer) and a grading rubric.
- Instead of a human just looking at it and saying "Looks good," the system checks specific points: "Did you label the lemming? Yes/No. Is the arrow pointing the right way? Yes/No."

3. The Grading System: The "Strict Teacher"

The paper introduces a very strict way of grading, which is the key innovation.

The "Strict" Score: This is like a professor who fails you for a single typo. If the AI draws a graph but labels the X-axis wrong, or misses one atom in a molecule, it gets a zero. It's all or nothing.
The "Relaxed" Score: This is like a teacher who gives partial credit. "You got the shape right, but the labels are messy. Here's a 60%."

4. The Results: The "Shock"

The researchers tested 17 different AI models (both the expensive, secret ones from big tech companies and the free, open-source ones).

The Good News: The top-tier, closed-source models (like Google's Nano Banana Pro) did okay. They got about 72% on the strict test. They are smart enough to understand the question.
The Bad News: Almost every other model failed miserably.
- Many open-source models got less than 3%.
- Some got 0%.
- The Analogy: Imagine a student who can recite the entire dictionary but fails to solve a simple math problem. The AI can generate a pretty picture of a molecule, but it doesn't understand chemistry. It draws the atoms in the wrong places because it's guessing, not reasoning.

5. Why This Matters

The paper argues that for AI to become truly "intelligent" (like a human expert), it can't just be a photocopier that makes pretty pictures. It needs to be a problem solver.

Current AI: "I see a prompt about a bridge. I will generate an image that looks like a bridge."
Future AI (The Goal): "I see a prompt about a bridge. I will calculate the physics, understand the materials, and draw a bridge that actually works."

The Bottom Line

GenExam is a wake-up call. It shows that while AI is getting amazing at making art, it is still terrible at doing science, engineering, and logic through images. It's like having a painter who can paint a perfect portrait of a surgeon, but if you ask them to actually perform the surgery, they have no idea what to do.

The paper releases this "exam" to the public so that researchers can try to fix this gap, pushing AI from being a "creative artist" to becoming a "true expert."

1. Problem Statement

Current benchmarks for Text-to-Image (T2I) generation primarily focus on:

Understanding and Reasoning: Evaluating text comprehension or visual reasoning (e.g., MMMU, MME).
General Aesthetics and World Knowledge: Assessing the ability to generate photorealistic images or illustrate simple concepts (e.g., GenEval, OneIG-Bench).

The Gap: There is a lack of rigorous evaluation for multidisciplinary exam-style image generation. Existing tasks often allow for "loose" interpretations of concepts. However, real-world expert tasks (like drawing a specific chemical reaction mechanism, a geometric proof, or a circuit diagram) require:

Strict Constraints: Precise adherence to complex, multi-step instructions.
Semantic Accuracy: The generated image must be factually correct according to domain knowledge (e.g., correct atom counts, correct graph topology).
Integration: The model must seamlessly integrate understanding, reasoning, and generation capabilities.
Evaluation Difficulty: Standard metrics (like CLIP score) or single-prompt MLLM judges fail to capture fine-grained errors (e.g., a missing bond in a molecule or a wrong axis label) that render an exam answer incorrect.

2. Methodology

A. The GenExam Benchmark

The authors introduce GenExam, the first benchmark dedicated to multidisciplinary text-to-image exams.

Scale & Scope: Contains 1,000 samples across 10 subjects: Mathematics, Physics, Chemistry, Biology, Computer Science, Geography, Economics, Engineering, Music, and History.
Taxonomy: Organized under a four-level hierarchical taxonomy (e.g., Mathematics/Plane Geometry/Circles/Tangent) based on UNESCO's ISCED-F standards to ensure academic rigor.
Prompt Design: Prompts are modeled after real exam questions (AP, A-level, IB), featuring complex, precise, and diverse constraints.
Ground Truth: Each sample includes a reference "Ground Truth" image and a set of fine-grained scoring points.

B. Evaluation Framework

Instead of relying on global similarity metrics, GenExam employs a scoring-point-based evaluation protocol using an MLLM-as-a-judge (specifically GPT-5).

Semantic Correctness:
- For each prompt, a set of specific "Yes/No" questions (scoring points) is generated (e.g., "Does the graph contain exactly 7 vertices?").
- The MLLM judge evaluates the generated image against these points, using the Ground Truth image as a reference.
- The score is the sum of weights for correct answers (range 0–1).
Visual Plausibility:
- Spelling: Correctness of text, notation, and equations.
- Logical Consistency: Coherence of marks, coordinates, and relationships (e.g., does the arrow point to the correct label?).
- Readability: Clarity of components, absence of occlusion, and proper layout.
- These are scored on a 0–2 scale.
Final Metrics:
- Strict Score: An image is "correct" only if Semantic Correctness = 1.0 AND all Plausibility scores = 2.0. This mimics the "all-or-nothing" nature of real exams.
- Relaxed Score: A weighted average of all dimensions to capture partial progress.

C. Data Curation

Collection: Web scraping and existing datasets (MMMU, ScienceQA, etc.).
Filtering: Automatic filtering via GPT-5 to remove low-resolution, non-English, or overly complex images.
Annotation: PhD annotators and GPT-5 collaborate to generate prompts and scoring points, ensuring the questions require subject knowledge and reasoning.

3. Key Contributions

First Multidisciplinary T2I Exam Benchmark: GenExam shifts the paradigm from "illustrating concepts" to "solving drawing exams," requiring deep domain knowledge integration.
Fine-Grained Evaluation Protocol: Introduces a rigorous, multi-dimensional scoring system (Semantic Correctness + Visual Plausibility) that detects subtle but critical errors often missed by global metrics.
Comprehensive Benchmarking: Evaluates 17 state-of-the-art models, including closed-source leaders (Nano Banana Pro, GPT-Image-1.5) and open-source T2I/Unified MLLMs.

4. Experimental Results

The experiments reveal a significant performance gap and the extreme difficulty of the benchmark:

Closed-Source Leaders: Even the best models struggle. Nano Banana Pro achieved the highest Strict Score at 72.7%, while GPT-Image-1.5 scored 43.2%.
Open-Source Models: Most open-source T2I and Unified MLLMs (e.g., FLUX.2 dev, Qwen-Image-2512, BAGEL) achieved Strict Scores below 5%, with many scoring 0% on specific subjects.
Failure Modes:
- Open-source models frequently fail at basic rendering (legibility, spelling, logical consistency) rather than just high-level reasoning.
- Closed-source models often get the structure right but fail on specific details (e.g., wrong number of atoms, missing edges in a graph).
Human Alignment: The MLLM-as-a-judge (GPT-5) shows high correlation with human expert evaluations (Pearson $r > 0.84$ ), validating the automated scoring protocol.

5. Significance and Future Directions

AGI Yardstick: GenExam serves as a crucial test for Artificial General Intelligence (AGI), as it requires the integration of understanding, reasoning, and generation—capabilities that current models lack in a unified manner.
Bottleneck Identification: The results suggest that the primary bottleneck for open-source models is not a lack of knowledge, but the inability to translate internal knowledge into accurate visual outputs (visual reasoning and execution).
Research Guidance: The benchmark highlights the need for models to improve:
- Precise text rendering and layout.
- Logical consistency in diagrams.
- The ability to follow strict, multi-constraint instructions in a domain-specific context.

In conclusion, GenExam establishes a new, rigorous standard for evaluating generative AI, demonstrating that while models can produce aesthetically pleasing images, they are far from capable of acting as expert-level "drawers" in multidisciplinary fields.