💻 computer science

PolyReal: A Benchmark for Real-World Polymer Science Workflows

The paper introduces PolyReal, a novel multimodal benchmark designed to evaluate Large Language Models on real-world polymer science workflows, revealing that while current models excel at theoretical reasoning, they significantly struggle with practical, context-dependent tasks like lab safety analysis and raw data extraction.

Original authors: Wanhao Liu, Weida Wang, Jiaqing Xie, Suorong Yang, Jue Wang, Benteng Chen, Guangtao Mei, Zonglin Yang, Shufei Zhang, Yuchun Mo, Lang Cheng, Jin Zeng, Houqiang Li, Wanli Ouyang, Yuqiang Li

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wanhao Liu, Weida Wang, Jiaqing Xie, Suorong Yang, Jue Wang, Benteng Chen, Guangtao Mei, Zonglin Yang, Shufei Zhang, Yuchun Mo, Lang Cheng, Jin Zeng, Houqiang Li, Wanli Ouyang, Yuqiang Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, all-knowing robot assistant. It has read every book in the library, memorized every textbook, and can chat fluently about history, math, and art. You ask it, "What is a polymer?" and it gives you a perfect, textbook definition.

But then, you take this robot into a real, messy chemistry lab. You hand it a photo of a cluttered workbench and ask, "Is this safe?" or show it a squiggly line on a graph from a machine and ask, "What does this mean?"

Suddenly, the robot stumbles. It might confidently tell you that a fire hazard is a "sparkly decoration" or invent a chemical reaction that never happened.

This is the story of PolyReal, a new "report card" designed to test exactly how well these AI robots handle the messy, real-world job of doing polymer science (the study of plastics, rubbers, and advanced materials).

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Textbook Robot" vs. The "Real-World Lab"

Current AI models are like students who ace every written exam but have never touched a beaker. They are great at reciting facts from their training data (like knowing the definition of "hydrophobicity"). However, when you put them in a real-world scenario where they need to look at a messy photo, read a raw data chart, or spot a safety danger, they fail.

The authors realized that existing tests were too easy. They were like asking the robot, "What is the capital of France?" (Easy). They needed a test that asked, "Here is a map of a city with a traffic jam; how do we reroute the buses?" (Hard, real-world).

2. The Solution: PolyReal (The "Polymer Science Obstacle Course")

The team created PolyReal, a benchmark that acts like a five-stage obstacle course for AI. Instead of just asking questions, it simulates the entire lifecycle of a real scientist's day:

Stage 1: The Knowledge Check (Foundational Knowledge): Can you apply what you know to a new situation?
- Analogy: You know the rules of soccer. Now, here is a muddy field with a broken goalpost. Can you explain how the game changes?
Stage 2: The Safety Inspector (Lab Safety): Can you look at a messy photo and spot the danger?
- Analogy: You walk into a kitchen where a gas stove is on, a bottle of bleach is next to ammonia, and a pile of paper is on the counter. Can you point out the three things that could cause a disaster?
Stage 3: The Detective (Mechanism Reasoning): Can you figure out how a chemical reaction works just by looking at a diagram?
- Analogy: You see a blueprint of a machine. Can you explain which gear turns which wheel and why the machine might jam?
Stage 4: The Data Translator (Raw Data Extraction): Can you read a messy, squiggly line from a machine and turn it into a clear answer?
- Analogy: A doctor gives you a scribbled, confusing X-ray. Can you tell them exactly what the bone looks like without making things up?
Stage 5: The Inventor (Application Exploration): Based on the material's properties, what can we actually build with it?
- Analogy: You have a new fabric that is waterproof but heavy. Should we make raincoats or boat anchors?

3. The Results: The "Capability Gap"

When they ran 15 of the smartest AI models through this obstacle course, they found a shocking gap:

The "Book Smart" Score: The models were great at Stage 1 (reciting facts). They got high marks for knowing the theory.
The "Street Smart" Score: The models crashed and burned at Stages 2, 4, and 5.
- The Safety Fail: They missed obvious dangers in photos because they were too busy "thinking" about the theory and not "looking" at the picture.
- The Hallucination Fail: When looking at raw data (like a graph), they would confidently invent facts. It's like looking at a blank page and saying, "I see a red line here," just to sound helpful.

The Big Takeaway: The AI is like a theoretical physicist who has never stepped foot in a lab. It knows the equations perfectly but can't handle the messy reality of the experiment.

4. Why This Matters

The authors argue that for AI to truly help scientists discover new medicines or materials, it can't just be a "search engine" that recites facts. It needs to be a reliable lab partner.

Current AI: "I read that polymers are strong. Therefore, this specific plastic must be strong." (Guessing based on text).
Future AI (What we need): "I see the stress test graph for this plastic. It snapped at 50 Newtons. Therefore, it is not strong enough for this bridge." (Reasoning based on evidence).

5. The "Hard" Truth

The paper also graded the questions by difficulty:

Easy: "What is this shape?" (AI is good at this).
Medium: "How do these two shapes fit together?" (AI is okay).
Hard: "Look at this messy lab photo with 10 items, identify the fire risk, explain why, and suggest a fix." (AI fails miserably).

The "Hard" questions are where the real science happens, and that is where current AI is still a novice.

Summary

PolyReal is a wake-up call. It tells us that while AI is amazing at reading books, it is still terrible at doing the actual work of science. To fix this, we need to train AI not just on textbooks, but on the messy, visual, and dangerous reality of real-world labs. Until then, we can't fully trust an AI to run a chemistry experiment on its own.

Technical Summary: PolyReal – A Benchmark for Real-World Polymer Science Workflows

1. Problem Statement

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in general domains but struggle significantly when applied to complex, real-world scientific workflows. Existing benchmarks in chemistry and materials science (e.g., MacBench, MatSciBench) primarily focus on isolated, atomic tasks (such as multiple-choice questions or simple chart reading) and often rely on closed-form evaluation formats. These benchmarks fail to capture the full lifecycle of scientific experimentation, which is a dynamic, multi-step, cross-modal process requiring the integration of foundational knowledge, safety analysis, mechanism reasoning, raw data extraction, and application exploration.

There is a critical gap in evaluating MLLMs on practice-based tasks where abstract scientific knowledge must be flexibly applied to unstructured, noisy, and context-dependent real-world data (e.g., cluttered lab scenes, raw instrument spectra, and safety protocols).

2. Methodology

2.1 Dataset Construction (PolyReal)

PolyReal is a novel multimodal benchmark comprising 545 high-quality question-answer pairs grounded in authentic polymer science research. The dataset was constructed through extensive collaboration with domain experts to ensure scientific rigor and authenticity.

Data Sourcing: Tasks were derived from real-world scenarios, including ongoing experiments, published papers, and authentic lab environments.
Scope: The dataset covers eight key sub-fields of polymer science (e.g., Bio-Functional, Photo-Functional, Nano-Functional, Smart Polymers).
Question Formats: Includes open-ended Q&A, numerical extraction, and specifically designed ranking tasks (34 items) to prevent shallow heuristics and force fine-grained comparative understanding.
Validation: All tasks underwent independent expert cross-validation to ensure scientific accuracy and alignment with module objectives.

2.2 Five-Stage Workflow Evaluation

PolyReal evaluates MLLMs across five independent modules that mirror the full polymer research lifecycle:

Foundational Knowledge Application (FKA): Applying core scientific principles to real-world scenarios (e.g., ranking hydrophobicity based on contact angle diagrams).
Lab Safety Analysis (LSA): Visual scene understanding and risk identification in cluttered, real-world lab photos (e.g., identifying fire hazards or obstructed fume hoods).
Experiment Mechanism Reasoning (EMR): Causal and procedural reasoning based on reaction diagrams and chemical structures.
Raw Data Extraction and Analysis (RDA): Parsing and digitizing key information from unstructured instrument outputs (e.g., NMR, IR spectra, .csv files).
Performance & Application Exploration (PAE): High-level reasoning to evaluate structure-property relationships and judge material suitability for specific applications.

2.3 Evaluation Protocol

Models Evaluated: 15 leading MLLMs, including top-tier closed-source models (O3, GPT-5, Gemini 2.5 Pro) and open-source models (Qwen3-VL, DeepSeek-R1).
Metrics: Precision (P), Recall (R), and F1-Score.
- Key Points Framework: Instead of binary accuracy, experts defined "Key Points" for each answer.
- Precision: Measures the absence of irrelevant or erroneous information (hallucinations).
- Recall: Measures the completeness of covering all correct Key Points.
Automated Evaluation: An LLM-as-a-Judge framework (using Gemini 2.5 Flash) was validated against human experts with high correlation (Spearman's $\rho$ > 0.91).
Setup: Zero-shot inference with Chain-of-Thought (CoT) enforcement to analyze reasoning processes.

3. Key Contributions

First Full-Workflow Benchmark: PolyReal is the first benchmark to comprehensively cover the entire polymer research lifecycle, moving beyond isolated tasks to evaluate integrated, multi-step scientific workflows.
Discovery of Capability Imbalance: The evaluation reveals a stark disconnect between knowledge-intensive reasoning and practice-based application. Models perform well on theoretical reasoning but fail significantly on tasks requiring visual context understanding and raw data parsing.
Diagnostic Analysis of Failure Modes: The paper provides a deep qualitative diagnosis of MLLM failures, identifying:
- Knowledge-Application Disconnect: Models can recite principles but fail to apply them to non-standard visual data.
- Factual Hallucination: Models invent non-existent spectral peaks or visual features to rationalize conclusions.
- Macromolecular Misunderstanding: Models often "downgrade" polymer concepts to small-molecule chemistry, missing unique properties like "site isolation."

4. Key Results

4.1 Overall Performance

Hierarchy: Closed-source models (O3, GPT-5) significantly outperform open-source models. O3 achieved the highest overall F1-score (0.601), while top open-source models (e.g., Qwen2.5-VL-72B) scored around 0.34.
Capability Imbalance:
- High Performance: Models excel in Foundational Knowledge Application (FKA) and Experiment Mechanism Reasoning (EMR), where training data (academic corpora) is abundant.
- Severe Deficiencies: Performance drops drastically in Lab Safety Analysis (LSA) and Raw Data Extraction (RDA). Even the best model (O3) scored only 0.412 F1 on LSA, indicating a failure to synthesize visual context with safety protocols.

4.2 Specific Failure Modes

High Recall, Low Precision: Models tend to generate verbose, "comprehensive" answers that cover correct points (High Recall) but introduce significant irrelevant or erroneous information (Low Precision).
Visual Perception Errors: In ranking tasks, models often misinterpret visual cues (e.g., confusing contact angle size with hydrophobicity) despite knowing the correct scientific principle.
Data Hallucination: In RDA tasks, models frequently hallucinate spectral peaks (e.g., claiming a "triplet" where none exists) to fit a pre-conceived molecular structure.

4.3 Ablation Studies

Modality Importance: Removing visual inputs caused a significant performance drop (e.g., O3 F1 dropped from 0.60 to 0.56), confirming that PolyReal is a genuine multimodal challenge.
External Knowledge: Providing search tools improved performance on knowledge-intensive tasks, suggesting that current models lack sufficient internal domain knowledge for complex scientific queries.
Difficulty Stratification: Performance degrades monotonically from Easy to Hard tasks. The "Hard" subset (involving holistic synthesis and safety analysis) acts as a critical discriminator, causing a "Recall Collapse" in many models.

5. Significance

PolyReal addresses a critical bottleneck in AI for Science. It demonstrates that while MLLMs have mastered abstract scientific knowledge, they lack the practical, context-dependent reasoning required for real-world scientific discovery.

Diagnostic Tool: It serves as a clear diagnostic for identifying specific weaknesses (e.g., hallucination in data parsing, failure in safety analysis) in current AI systems.
Roadmap for Improvement: The findings suggest that future AI agents for science must be trained on multimodal, unstructured, real-world data and developed with mechanisms to prevent "defensive verbosity" and factual hallucinations.
Benchmark Standard: It sets a new standard for evaluating scientific AI, shifting the focus from "what the model knows" to "how the model applies knowledge in complex, real-world scenarios."