💬 NLP

Can Language Models Imagine Without Seeing? Ekphrasis: Measuring Visual Creative Ideation in Text-Only LLMs

This paper introduces Ekphrasis, a novel benchmark and evaluation framework that isolates and measures the visual creative ideation capabilities of text-only language models by distinguishing between fluent prose and the actual ability to generate useful, expressive, and population-novel visual plans.

Original authors: Hongyu Luo, He Wang, Huihao Jing, Hong Ting Tsang, Yuxuan Liu, Wuganjing Song, Yauwai Yim, Chunyang Li, Yangqiu Song

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hongyu Luo, He Wang, Huihao Jing, Hong Ting Tsang, Yuxuan Liu, Wuganjing Song, Yauwai Yim, Chunyang Li, Yangqiu Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director trying to cast a movie, but you can't see the actors yet. You only have a scriptwriter who describes the scenes in words. The big question is: Can this scriptwriter actually imagine a new, exciting scene, or are they just stringing together fancy words that sound creative but describe the exact same old clichés everyone has seen a thousand times? This is the puzzle at the heart of a new study in the world of Artificial Intelligence, specifically looking at "Large Language Models" (LLMs). These are the super-smart computer programs that write stories, answer questions, and chat with us. Scientists have long wondered if these text-only machines can truly "visualize" ideas before they are turned into pictures, or if they are just faking it with flowery language. To find out, researchers needed a way to test if an AI's description was a genuine spark of visual invention or just a recycled, safe idea dressed up in a tuxedo.

Enter EKPHRASIS, a new "exam" designed by researchers at The Hong Kong University of Science and Technology to test exactly this. Think of EKPHRASIS as a creative gym for AI. Instead of asking the AI to draw a picture (which it can't do directly in this test), the researchers gave 14 different AI models 400 specific creative challenges. These challenges asked the models to do things like turn an abstract concept like "time pressure" into a visual scene, mash two unrelated ideas together, or rewrite an object in a new way. The goal wasn't just to see if the AI wrote a pretty sentence, but to see if it could come up with a Visual Creative Ideation (VCI) plan: a description that is useful for the task, expressive enough to paint a clear picture in your mind, and, most importantly, novel—meaning it didn't just repeat the boring, overused ideas that other AIs always come up with.

The researchers discovered that fluent, beautiful writing is a bit of a trap. An AI can write a paragraph that sounds incredibly creative and vivid, yet be describing the exact same old visual clichés, like using an hourglass to represent time or a storm cloud to represent sadness. To catch this "novelty illusion," the team built a special scoring system. They didn't just ask, "Is this creative?" They asked, "Is this useful? Is it easy to picture? And is it different from what the other 13 AIs would have said?" They used a clever method called "Typed Idea Graphs" to map out the common, boring patterns that all the models tend to repeat, and then checked if the new answers avoided those traps.

The results were fascinating and nuanced. The study found that being "good" at visual ideation isn't just one single superpower. Some AIs were great at following instructions and making useful plans but were very clichéd. Others were wildly original but sometimes missed the point of the task. The top performers, like Gemini 3.1 Pro and GPT-5.4, managed to balance all three traits, but they did it in different ways. Crucially, the researchers took the best text descriptions and actually generated images from them using a separate tool. When humans looked at these images without seeing the original text, they still preferred the images that came from the "high VCI" text plans. This suggests that the AI really was planning something visual, not just writing pretty prose.

However, the paper is careful to note that this doesn't mean the AI has a human-like "mind's eye" or private imagination. It simply means that these text-only models can create detailed, image-ready blueprints that survive the journey from text to picture. The study suggests that while current models are getting better at visual planning, they still struggle to break free from the "safe" ideas that their training data teaches them to love. The takeaway is clear: just because an AI writes a creative story doesn't mean it has a creative visual idea. To truly measure visual creativity, we have to look past the flowery words and check if the blueprint is actually new, useful, and ready to be built.

Technical Summary: EKPHRASIS – Measuring Visual Creative Ideation in Text-Only LLMs

Problem Statement

Current evaluations of Large Language Models (LLMs) often fail to isolate whether text-only models can genuinely originate visual concepts before image generation occurs. A significant "measurement trap" exists where fluent, vivid visual prose can mask a lack of genuine visual invention. Models may produce outputs that appear creative but rely on familiar visual clichés, safe motifs, or recurring population patterns (the "novelty illusion"). Existing benchmarks typically evaluate final rendered images, verbal creativity, or instruction following, but do not assess the pre-image visual plan—the textual blueprint that must carry visual substance, specificity, and novelty before pixels exist.

The core question addressed is: Can language models imagine visually, or only write as if they can?

Methodology: EKPHRASIS Benchmark

The authors introduce EKPHRASIS, a 400-task benchmark designed to measure Visual Creative Ideation (VCI). VCI is defined as the ability to produce textual visual plans that are:

Useful: Satisfying the specific visual brief.
Expressive: Providing enough detail to support a concrete, stable mental image.
Novel: Departing from task-specific visual clichés common in the LLM population.

Task Design

The benchmark spans four controlled task families, each testing a different aspect of visual planning:

Abstraction: Mapping abstract concepts to concrete visual carriers (e.g., expressing "time pressure" without using an hourglass).
Combination: Fusing two distinct concepts into a single, coherent visual entity rather than a simple juxtaposition.
Transformation: Rewriting an object's form, material, or context under specific constraints while maintaining recognizability.
Adaptation: Designing a visual solution for a real-world context (e.g., advertising, education) with specific audience and medium constraints.

Evaluation Protocol

EKPHRASIS employs a rigorous, multi-stage evaluation pipeline to avoid biases associated with fluency or length:

Two-Field Response Format: Models must output a Core Idea (carrier, metaphor, mechanism) and a Visual Description (objects, layout, material, light). This prevents fluent prose from replacing a specific visual plan.
Checklist-Assisted Pairwise Comparison: Human and LLM judges compare anonymized model pairs using dimension-specific checklists for Usefulness, Expressiveness, and Novelty.
Typed Idea Graphs (TIG) for Novelty: To measure novelty objectively, the system constructs a "population-anchored" reference. For each task, a graph of "typed idea atoms" (concepts, motifs, mechanisms) is built from the responses of all models except the two being compared (Leave-Two-Models-Out). High-scoring clusters in this graph represent "population clichés." Novelty is scored based on whether a response avoids, transforms, or repeats these specific clichés.
Bradley-Terry Aggregation: Pairwise preferences are aggregated using Bradley-Terry models to generate latent skill scores ( $\theta$ ) for each model across the three dimensions, with active sampling used to optimize the comparison budget.

Cross-Modal Validation

To ensure text-level scores reflect actual visual content rather than just verbal quality, the authors conducted a cross-modal grounding study:

Textual plans were rendered into images using a fixed text-to-image system.
A "faithfulness gate" filtered out renderings that failed to preserve the source idea.
Human annotators compared the rendered images without seeing the original text or model identity.
The resulting image-level preferences were compared against the original text-level VCI rankings to verify convergent validity.

Key Contributions

Construct Definition: The paper operationalizes Visual Creative Ideation (VCI) as a distinct construct separating brief satisfaction, visual specificity, and population-novelty from general fluency or verbal creativity.
EKPHRASIS Benchmark: A 400-instance suite covering Abstraction, Combination, Transformation, and Adaptation, featuring a novel evaluation engine that uses Typed Idea Graphs to anchor novelty against recurring model population patterns.
Empirical Evidence: Evaluation of 14 state-of-the-art text-only LLMs demonstrating that VCI is multidimensional. Strong models achieve high overall scores through different profiles (e.g., some excel in Usefulness but lack Novelty, while others are highly Novel but less Useful).
Validation: Evidence that text-level VCI ordering largely survives faithful rendering and blind image-level preference judgment, supporting the metric as a measure of visual ideation beyond prose quality.

Results

Model Performance: Top-tier models (Gemini 3.1 Pro, GPT-5.4, Claude 4.7, DeepSeek V4, Kimi K2.6) form a diverse group. For instance, GPT-5.4 led in Usefulness (+1.53), Claude 4.7 and Gemini 3.1 Pro in Expressiveness, and Kimi K2.6 in Novelty (+1.66).
Dimensional Dissociation: The study confirms that high usefulness does not guarantee visual novelty. Models like Grok 4.3 showed high Usefulness (+1.10) but below-average Novelty (-0.17), indicating they can satisfy briefs while relying on familiar visual carriers.
Task Difficulty: Combination tasks were identified as the most demanding, requiring shared visual structure rather than side-by-side placement.
Novelty Ablation: Replacing task-specific TIG anchors with generic novelty references reduced human-majority agreement by 20.9 percentage points, proving that task-specific population anchoring is critical for accurate novelty measurement.
Cross-Modal Agreement: In the faithful subset, 96.6% of decisive image-level preferences aligned with the text-level model ordering, validating that the text plans captured recoverable visual information.

Significance and Claims

The paper claims a "qualified operational yes" to the question of whether text-only models can imagine visually. It demonstrates that:

Pre-image plans can carry task-relevant visual information across a text-to-image modality shift.
VCI is not monolithic; similar aggregate scores can mask different underlying capabilities (e.g., brief following vs. genuine visual invention).
The central boundary exposed is "safe plausibility": many outputs satisfy briefs and read well but fall back on familiar carriers and stock metaphors.
Practical Implication: Creative-looking prose should not be treated as visual creativity without separate checks for visual mechanism, imageability, and population novelty.

The authors modestly note limitations, including dependence on the current model population for defining clichés, cultural assumptions in novelty judgments, and the reliance on a specific renderer for cross-modal validation. They emphasize that EKPHRASIS measures externalized pre-image planning, not private mental imagery or a general theory of human creativity.