💬 NLP

GENIE: A Fine-Grained Measure for Novelty

This paper introduces GENIE, a fine-grained evaluation metric designed to measure the novelty of large language model responses along task-specific features, addressing the limitations of holistic metrics and providing insights into the effectiveness of creativity mitigation methods.

Original authors: Ramya Namuduri, Manya Wadhwa, Anshun Asher Zheng, Greg Durrett, Junyi Jessy Li

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ramya Namuduri, Manya Wadhwa, Anshun Asher Zheng, Greg Durrett, Junyi Jessy Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a talent show where 100 contestants are asked to tell a story about a dinosaur and a computer.

Most of the stories sound exactly the same: a T-Rex finds a laptop in a jungle, gets scared, and runs away. But then, one contestant tells a story about a T-Rex who is actually a bored office worker trapped in a prehistoric body, trying to fix a broken toaster on a boat.

The Problem with Old Judges
In the past, if you wanted to know how "creative" or "new" a story was, you used a "holistic" judge. This judge would look at the whole story and give it a single score, like "7 out of 10."

The Flaw: This is like a teacher giving you a single grade for a math test without telling you which problems you got right or wrong. If the dinosaur story got a "7," you wouldn't know if it was creative because of the setting (the boat), the character (the office worker), or the plot.
The Confusion: Sometimes, a story might look new just because the writer used fancy words (paraphrasing), but the actual ideas are boring. Old judges often get tricked by these fancy words.

The New Solution: GENIE
The authors of this paper built a new tool called GENIE (Granular Evaluation of Novel Ideas with Explainability). Think of GENIE not as a single judge, but as a team of specialized inspectors.

Instead of giving one score, GENIE breaks the story down into specific "features" or categories, like:

Setting: Where does it happen?
Plot: What actually happens?
Character: Who is the main person (or dinosaur)?
Style: How is it written?

How GENIE Works (The "Question & Answer" Game)
GENIE doesn't just guess; it plays a game of "Twenty Questions" to understand the story:

The Population: First, GENIE looks at a huge pile of other stories (the "population") to see what the "standard" dinosaur stories look like.
The Questions: For a new story, GENIE asks specific questions based on the features.
- Question: "What is the setting?"
- Answer: "A boat."
The Comparison: GENIE checks the answer against the pile of other stories.
- If 99 other stories said "Jungle," and this one says "Boat," GENIE gives a high score for Setting Novelty.
- If 99 other stories said "The dinosaur finds a computer," and this one says the same thing, GENIE gives a low score for Plot Novelty.

Why This is Better
The paper claims that GENIE is much better at spotting the truth than the old "holistic" judges.

It's Honest: It can tell you exactly where the creativity is. "This story is very new in its setting, but very boring in its plot."
It's Not Fooled by Fancy Words: If a writer just rewrites a boring story with different synonyms (paraphrasing), the old judges might think it's new. GENIE sees that the answers to its questions are still the same, so it knows the story isn't actually new.
It Separates "New" from "Good": Sometimes a story is weird and new but terrible to read. Old judges often mix "creativity" with "quality." GENIE separates them, measuring only how different the ideas are, not how well they are written.

Testing the Tool
The authors tested GENIE by:

Surgery: They took a boring story and surgically changed just one thing (like changing the setting from a jungle to a boat). GENIE correctly spotted that only the setting changed and gave a high score for that specific part.
Paraphrasing: They took a story and rewrote it with different words but the same meaning. GENIE correctly said, "This isn't actually new," while the old judges got confused and thought it was different.

The Bottom Line
GENIE is a new way to measure creativity that doesn't just give a vague score. It acts like a microscope, showing us exactly which parts of a story are fresh and which parts are just copies of what everyone else is doing. This helps researchers understand not just if AI is creative, but where it is creative and where it needs to improve.

Technical Summary: GENIE: A Fine-Grained Measure for Novelty

Problem Statement

Large Language Models (LLMs) have consistently demonstrated a lack of creativity and diversity in their outputs. While prior work has established that models struggle to generate diverse outputs using lexical-semantic approaches, creativity tests, and human experts, existing evaluation methods fail to isolate precisely where models lack creativity on specific tasks.

Current metrics for novelty and creativity are largely holistic, producing a single scalar score. These approaches struggle to capture the high-dimensionality of novelty and fail to explain why a response is considered novel or not. For instance, a response might be unique in its setting but mundane in its plot, yet holistic metrics cannot distinguish these fine-grained differences. Furthermore, many existing metrics conflate novelty with other components of creativity, such as quality or aesthetic value, or rely on implicit reference points (e.g., pretraining corpora) rather than explicit populations.

Methodology: The GENIE Framework

The authors propose GENIE (Granular Evaluation of Novel Ideas with Explainability), a framework designed to quantify the fine-grained nuances of novelty by extracting task-specific features relative to a user-defined population of responses.

Core Components

Feature Discovery: Instead of manually defining features, GENIE automatically discovers task-specific axes (features) by prompting an LLM to generate feature descriptions, which are then clustered to form a set $F = \{f_1, ..., f_N\}$ .
Question Generation and Mapping: To ground these features to specific prompts, GENIE generates a set of questions $Q$ for a given prompt $p$ . These questions are mapped to the discovered features. This process utilizes the "alternative semantics of questions" view, where questions partition possible worlds into subspaces corresponding to feature values.
Population-Based Scoring: GENIE measures novelty relative to an explicit population $D_{pop}$ $D_{p o p}$ of documents generated by various models.
- For a target document $d_t$ , the system answers the mapped questions to extract feature values.
- It calculates the dissimilarity between the target's answers and the answers from the population documents for each question.
- The novelty score for a feature $f$ is the average pairwise dissimilarity across all questions mapped to that feature.
Output: Unlike holistic metrics, GENIE outputs a vector of novelty scores, one for each feature (e.g., Setting, Plot, Character), providing explainability regarding which specific aspects of a response are novel.

Implementation in Creative Writing

The paper instantiates GENIE in the domain of long-form story generation.

Features: Through automatic discovery and clustering, the authors identified features corresponding to established taxonomies: Agent, Perspective, Plot, Setting, Social Atmosphere, and Style.
Data: The study utilized ~10,000 documents generated by 40 models across 50 open-ended prompts. A population of 3,404 documents (from 21 models) was constructed to serve as the reference, while 18 target models were evaluated against this population.
Dissimilarity: Similarity between answers was judged using Qwen2.5-32B-Instruct on a 4-point Likert scale, converting similarity to dissimilarity ($1 - similarity$).

Key Results

1. Sensitivity and Robustness

The authors conducted causal intervention studies to test GENIE's validity:

Sensitivity: When specific features (e.g., Plot or Setting) were surgically edited in documents, GENIE successfully detected statistically significant changes in the novelty scores for those specific features.
Robustness: When documents were paraphrased (semantically equivalent but lexically different), GENIE showed no significant change in scores, demonstrating robustness against surface-level variations.
Comparison: In contrast, holistic metrics like Cosine Distance and NoveltyBench were found to be either lexically sensitive (failing the robustness test) or capturing extraneous properties. For example, Cosine Distance often failed to distinguish between different types of content interventions (e.g., Plot vs. Style) and was sensitive to paraphrasing.

2. Evaluation of Mitigation Methods

GENIE was used to evaluate methods designed to improve model creativity (e.g., Verbalized Sampling, Outlining, Negative Sampling, Conformative Decoding).

Fine-Grained Insights: While holistic metrics might suggest a method improves "creativity" generally, GENIE revealed that specific methods improve specific features. For instance, Outlining significantly improved Perspective and Style, while Conformative Decoding improved Setting.
Artifact Detection: The framework highlighted that some methods (like Verbalized Sampling) produced responses that were disproportionately short, which skewed novelty scores when compared to a population of full-length stories. This underscores the importance of population alignment.

3. Intrinsic Validation

Human studies validated the pipeline components:

Question Generation: Achieved moderate-to-high agreement with human annotators regarding question validity.
Feature Mapping: Showed high accuracy in mapping generated questions to the correct features.
Similarity Judgments: The LLM used for dissimilarity scoring (Qwen2.5-32B) showed strong correlation with human consensus, validating the automated scoring mechanism.

Significance and Claims

The paper claims that GENIE provides a diagnostic lens into the evaluation of LLM creativity. Its primary significance lies in:

Explainability: It moves beyond "black box" scalar scores to identify which task-specific properties (e.g., plot vs. setting) contribute to a response's novelty.
Decoupling Novelty from Quality: By tethering feature extraction to the task and measuring relative dissimilarity, GENIE isolates novelty from aesthetic value or quality, addressing a limitation in metrics that conflate these dimensions.
Domain Agnosticism: While demonstrated on creative writing, the framework is designed to be domain-agnostic, capable of adapting to scientific discovery or other tasks by redefining the feature set.
Improved Mitigation Analysis: It enables a more precise understanding of how creativity mitigation methods work, revealing that they often target specific dimensions of output rather than creativity as a monolithic concept.

The authors acknowledge limitations, noting the computational cost of the pipeline (requiring multiple LLM calls for feature discovery, question generation, and scoring) and that the current validation is primarily within the creative writing domain. They do not claim the method is perfect for all domains without further validation but posit that the cost is justified to move beyond coarse, surface-level proxies.