💬 NLP

Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation

This paper introduces SciStyleBench, a comprehensive benchmark demonstrating that LLM judges are significantly biased by writing style rather than scientific substance, and proposes SciStyleExtractor as an effective module to mitigate this bias while improving the accuracy of idea evaluation.

Original authors: Fengxian Ji, Yuke Li, Jingpu Yang, Juanfan Wu, Fan Zhang, Zhexuan Cui, Yu Xie, Min Peng, Qianqian Xie, Xiuying Chen, Zhuohan Xie

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fengxian Ji, Yuke Li, Jingpu Yang, Juanfan Wu, Fan Zhang, Zhexuan Cui, Yu Xie, Min Peng, Qianqian Xie, Xiuying Chen, Zhuohan Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where super-smart computers are helping scientists come up with new ideas for curing diseases, building faster rockets, or understanding the stars. These computers, called Large Language Models (or LLMs for short), are like tireless brainstorming partners. They can spit out thousands of potential scientific ideas in the time it takes a human to brew a cup of coffee. But here's the catch: we can't test every single one of those ideas in a real lab. It's too expensive and takes too long. So, we need a way to pick the "winners" from the crowd.

This is where the "Judge" comes in. Think of an LLM-as-Judge as a digital talent scout. Its job is to read through these thousands of computer-generated ideas and decide which ones are truly brilliant and which ones are just fluff. The big question everyone is asking is: Is this digital scout actually looking at the quality of the idea, or is it just getting distracted by how the idea is written? It's a bit like a music competition where the judges might give a higher score to a singer who wears a flashy costume and speaks with a confident voice, even if their voice is actually off-key, while ignoring a talented singer who wears a plain t-shirt and stutters a bit. If the judges are fooled by the "style" instead of the "substance," we might end up funding the wrong projects and missing out on real breakthroughs.

This is exactly what a team of researchers set out to investigate in their new paper, titled "Style Wins, Substance Loses." They wanted to see if these AI judges are fair or if they are easily tricked by fancy wording. To do this, they built a special testing ground called SciStyleBench. Imagine a giant science fair where they take the exact same scientific idea—say, a plan to grow plants on Mars—and rewrite it in fifteen different ways. Some versions are super short and boring, some are long and full of exciting adjectives, some sound overly confident, and some are written like a grand movie trailer. Crucially, the science inside all these versions is identical; only the "clothing" changes.

When they let the AI judges evaluate these ideas, the results were a bit alarming. The judges were like fashion-conscious critics who couldn't see past the outfit. They tended to give higher scores to the ideas that sounded more confident, used more complex words, or had a "grand narrative," even though the actual science was the same as the boring versions. In fact, when they changed the style, the ranking of the ideas shifted dramatically. An idea that was in the top 5 when written plainly could drop out of the top 30 just because it was rewritten to sound more "hype-y." The paper suggests that current AI judges are surprisingly sensitive to these superficial tricks and not very good at spotting the real scientific value underneath.

But the researchers didn't just stop at pointing out the problem; they tried to build a fix. They created a new tool called SciStyleExtractor. Think of this as a "style translator" or a "truth detector" that sits in front of the judge. Before the judge reads the idea, this tool analyzes the text, figures out what kind of "clothing" it's wearing (e.g., "Oh, this one is trying to sound super confident"), and then whispers a note to the judge: "Hey, don't let the confidence fool you; look at the actual science."

When they tested this new setup, the results improved significantly. The AI judges became much better at ignoring the flashy style and focusing on the real scientific meat. They stopped being fooled by the "hype" and started recognizing the truly valuable ideas much more often. The paper shows that while we can't make the judges perfectly immune to style yet, adding this "style translator" makes them much fairer and more reliable. The main takeaway is that for AI to truly help science, we need to teach it to look past the shiny packaging and judge the gift inside. If we don't, we risk letting the loudest, flashiest ideas win, while the quiet, brilliant ones get left behind.

Technical Summary: Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation

Problem Statement

The rapid proliferation of AI-driven scientific agents has shifted scientific idea generation from a writing assistance task to a core component of automated research workflows. While Large Language Models (LLMs) can generate vast numbers of candidate ideas at low cost, downstream processes—such as experimental validation, literature search, and human review—require substantial resources. Consequently, reliable evaluation mechanisms are critical for filtering ideas.

Current evaluation relies heavily on LLM-as-Judge systems. However, a fundamental gap exists: it remains unclear whether these judges evaluate the scientific substance of an idea or are unduly influenced by superficial stylistic presentation. Existing benchmarks often entangle content with style, lack controlled environments for style perturbation, and fail to provide formal metrics to quantify stylistic bias. Furthermore, standard evaluation methods (e.g., pairwise ranking, multi-dimensional scoring) do not explicitly model stylistic "nuisance factors," leading to a situation where "Style Wins, Substance Loses."

Methodology: SciStyleBench

To address these gaps, the authors propose SciStyleBench, a unified framework comprising three core components designed to diagnose and mitigate stylistic bias.

1. SciStyleStage: A Controlled Evaluation Environment

This component establishes a style-paired benchmark to isolate the effects of presentation style from scientific substance.

Data Construction: The dataset includes 600 scientific ideas sourced from NeurIPS and ICLR across diverse domains (CS, Biology, Math, Astronomy, Economics).
Perturbation Space: Each idea is transformed into 15 controlled variants using DeepSeek-V3. These variants are categorized into:
- Single-Style Variants (11 types): Including baselines (Identity, Paraphrase, Plain Core) and specific style manipulations such as Verbose (expanded detail), Grand Narrative (visionary framing), Overconfident (assertive tone), Novelty Emphasis, Application Framing, Hollow (persuasive but empty), Flawed (logical weaknesses), and Enriched (substantive additions).
- Mixed-Style Variants (4 types): Combinations like Deceptive Hollow (visionary framing + hollow content) and Ultimate Hype.
Three-Stage Evaluation: To test robustness across evidence contexts, ideas are evaluated under:
1. No Background: Relying solely on the idea text.
2. Fixed-Domain Context: Using a static pool of domain-specific literature.
3. Idea-Specific Retrieval: Using dynamically retrieved papers relevant to the specific idea.

2. SciStyleMetrics: Quantitative Bias Measurement

The framework introduces three metrics to characterize judge performance:

Style Bias Index (SBI): Measures score stability when scientific substance is fixed but style changes. A lower SBI indicates better invariance to style.
Substance Recognition Rate (SRR): Measures the judge's ability to correctly rank ideas where one has demonstrably stronger substance (e.g., Enriched vs. Plain vs. Hollow). A higher SRR indicates better sensitivity to scientific merit.
Adversarial Win Rate (AWR): Measures whether the judge prioritizes substance over style when they conflict (e.g., a high-substance plain idea vs. a low-substance highly persuasive idea). A higher AWR indicates robustness against adversarial stylistic packaging.

3. SciStyleExtractor: A Mitigation Module

To reduce bias, the authors propose SciStyleExtractor, a plug-and-play auxiliary module.

Mechanism: Instead of rewriting the idea, the extractor predicts style-related nuisance signals ( $z_p$ ) before the main judge evaluates the text. These signals include the style type, deviation from neutral presentation, and a bias-control instruction.
Training: The extractor is trained via Teacher-Student Distillation. A "Teacher" judge (provided with ideal style signals) guides a "Student" judge (using predicted signals) to align their output distributions, minimizing the Kullback-Leibler (KL) divergence.
Inference: The trained extractor injects structured style information into a frozen judge, enabling the judge to focus on substance while being aware of stylistic distortions.

Key Results

Experiments were conducted on 27,000 evaluation instances across six judge models (including general-purpose LLMs and scientific-review specialized models) under the three background settings.

Direct Judges are Sensitive to Style: Standard "Direct" judges exhibited high SBI (0.566), indicating significant score fluctuations based on style. They also showed weak substance discrimination (SRR: 0.504) and low adversarial robustness (AWR: 0.554).
Context Does Not Fully Solve the Problem: While adding retrieval context reduced SBI slightly, it did not consistently improve SRR or AWR, suggesting that external evidence alone cannot disentangle style from substance.
Low SBI $\neq$ Robustness: Some models achieved near-zero SBI but also near-zero SRR and AWR, indicating "score collapse" (assigning identical scores to all ideas) rather than genuine robustness.
SciStyleExtractor Improves Performance: The trained extractor significantly improved the balance of metrics:
- SBI decreased from 0.566 to 0.501.
- SRR increased from 0.504 to 0.759.
- AWR increased from 0.554 to 0.899.
- The extractor achieved this by improving substance sensitivity and adversarial robustness without simply flattening scores.
Ranking Instability: Despite improvements, residual ranking instability remains. Style-induced changes caused ideas to enter or leave the Top-K selection set (e.g., 15.9% change rate at Top-30), highlighting the risk of resource misallocation in automated research pipelines.

Significance and Claims

The paper claims that robust scientific idea evaluation requires invariance to stylistic variation without sacrificing sensitivity to scientific substance.

Systematic Framework: SciStyleBench provides the first systematic framework for identifying, quantifying, and mitigating stylistic bias specifically in the context of scientific idea generation.
Diagnostic Capability: It reveals that current LLM-as-Judge systems are heavily influenced by rhetorical presentation (verbosity, confidence, framing) rather than methodological soundness.
Practical Mitigation: The study demonstrates that explicit style-awareness via an auxiliary extractor module can significantly improve evaluation reliability, offering a path toward more trustworthy automated research workflows.
Nuanced Conclusion: The authors modestly note that while SciStyleExtractor improves the balance of metrics, it does not completely eliminate style-induced ranking shifts, particularly for hybrid variants that combine degraded substance with persuasive presentation.

The work concludes that future evaluation systems must explicitly decouple presentation style from scientific content to ensure that downstream research resources are allocated based on genuine scientific merit.