🔬 materials science

Separating perception from reasoning in vision-language models: a model-free render ceiling for crystal structures

This paper introduces the "render ceiling," a model-free benchmarking method that uses inverted camera rendering of known crystal structures to definitively separate visual perception errors from reasoning failures in vision-language models, revealing that many models struggle with geometric extraction rather than logical inference.

Original authors: Can Polat, Mustafa Kurban, Erchin Serpedin, Hasan Kurban

Published 2026-09-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Can Polat, Mustafa Kurban, Erchin Serpedin, Hasan Kurban

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of materials science, researchers spend their days trying to understand how atoms arrange themselves to form the solids that make up our world. These arrangements, known as crystal structures, are often visualized as three-dimensional drawings where spheres represent atoms and sticks represent the bonds holding them together. For decades, scientists have relied on these images to communicate complex spatial relationships, but a new generation of artificial intelligence is now being asked to read them. These vision-language models are designed to look at an image and answer questions about it, acting as a bridge between what a computer sees and what a human understands. The hope is that these systems could eventually help design new materials by interpreting scientific figures just as a human researcher would. However, a persistent problem has made it difficult to know if these machines are truly learning to see or if they are simply guessing based on patterns in the text. When a model gets an answer wrong, it is nearly impossible to tell if it failed because it could not interpret the picture correctly or because it failed to use logic to solve the puzzle. Without a way to separate these two skills, the scientific community cannot know where to focus its efforts to improve these tools.

A team of researchers has now introduced a new way to measure these models that bypasses the need for guesswork entirely. Instead of relying on another computer program to act as a judge, they created a system based on the exact mathematical rules used to draw the images in the first place. They started with a known set of crystal structures, whose atomic positions were perfectly defined, and used a standard camera setup to generate thousands of images. Because they knew exactly how the camera was positioned and what the original object looked like, they could mathematically reverse the process. By inverting the camera angles and re-calculating the positions of the atoms from the different views, they could recover the original structure with perfect precision. This process created a "ceiling" of perfect accuracy, a benchmark that represents the absolute maximum amount of information available in the images. If the images themselves contain the answer, this method proves it. If a model fails to find the answer, the failure must belong to the model, not the picture.

The researchers tested this method on over two thousand crystal structures, checking to ensure that no two atoms ever appeared to overlap in a way that would confuse the calculation. They found that for every single one of these structures, the mathematical reversal worked perfectly, recovering the correct answer every time. This meant the ceiling was solid; the images held the full truth. They then asked fourteen different vision-language models to identify the crystal system of these same images. The results were revealing. While the models performed better when they were given the exact atomic coordinates as text, they still fell far short of the perfect score, even with that extra help. For most of the models, the gap between their performance and the perfect ceiling was not caused by an inability to see the image. Instead, the majority of their errors happened after they had supposedly understood the picture, during the reasoning phase. The study showed that for thirteen out of the fourteen models, the difficulty lay in their logic, not their vision.

To prove that the images themselves were readable, the researchers trained a simple computer vision system that had no language capabilities at all. This system looked only at the pixels of the images, without any text or chemical knowledge. Remarkably, this simple system scored higher than every single vision-language model tested, correctly identifying the crystal structures in nearly ninety percent of cases. This finding suggests that the images are not too complex for a machine to read; rather, the complex models are struggling to process the information they see. The study also uncovered a hidden flaw in how these systems work. When asked to extract the coordinates of atoms from the images, a powerful model produced lists of numbers that looked perfectly formatted but were completely wrong, matching almost none of the actual atoms in the picture. In a standard test, this kind of error would be blamed on poor reasoning, but this new method showed it was actually a failure to see the image at all. The model was fabricating data, creating a false reality that looked correct on the surface but was empty underneath.

The implications of this work extend beyond just grading test scores. The researchers demonstrated that the way cameras are placed around an object matters more than the number of cameras used. They found that a specific arrangement of five views was enough to eliminate any ambiguity, while fewer views left room for error. This provides a clear rule for scientists who create benchmarks: the quality of the view matters more than the quantity. More importantly, the study offers a new tool for auditing artificial intelligence in scientific workflows. In the future, as these models are used to design new materials, this method can act as a check to ensure that the intermediate steps of their reasoning are based on real data and not on hallucinated numbers. By separating what the machine sees from how it thinks, the researchers have provided a way to build more reliable tools for science, ensuring that when a model claims to understand a crystal, it is actually seeing it.

Technical Summary: Separating Perception from Reasoning in Vision–Language Models via a Model-Free Render Ceiling

Problem Statement
Current multimodal evaluation methods struggle to distinguish whether a vision–language model (VLM) fails due to perception (misreading the visual input) or reasoning (misinterpreting correct visual information). Existing approaches to separate these stages invariably introduce a second model into the evaluation loop (e.g., using a model to generate textual descriptions of images or using ensembles to estimate a "best-case" score). Consequently, the reference against which a model's deficit is measured is itself a model, inheriting its own errors and obscuring the true information content of the image. This is particularly acute in materials science, where models tasked with reading crystal structure renders often fail at spatial reasoning and cross-modal synthesis, yet the gap between model performance and the theoretical limit of the image remains unmeasured.

Methodology: The Render Ceiling
The authors introduce a model-free render ceiling, a reference benchmark constructed by rendering known objects (crystal structures) and mathematically inverting the process to recover the exact answer the images support.

Forward Rendering: Crystal structures (lattice, fractional coordinates, species) are rendered into five orthographic views using a fixed, known camera protocol.
Geometric Oracle (Inversion): Instead of analyzing pixels, the authors invert the frozen cameras. They back-project rays from the known 2D image coordinates, re-solve cross-view correspondence via ray intersection (triangulation), and merge the resulting 3D points.
Certification: The reconstructed 3D positions are fed into a deterministic symmetry algorithm (spglib) to assign the crystal system.
- The Phantom Set ( $\Phi_\delta$ ): The only failure mode of this oracle is a "projection coincidence" where distinct atoms project to the same location within a tolerance $\delta$ , creating a "phantom" atom. The authors prove that for their specific protocol and sample, this set is empty.
- The Ceiling ( $R_1$ ): Because the phantom set is empty, the oracle achieves 100% accuracy ( $R_1 = 1.0000$ ). This establishes a certified, model-free upper bound: every point of a VLM's deficit is attributable to the model, not the image.
Attribution Ladder: The authors evaluate 14 VLMs across three rungs:
- $R_4$ : Accuracy on pixel renders (Perception + Reasoning).
- $R_3$ : Accuracy when ground-truth geometry is provided as text (Reasoning only).
- $R_1$ : The model-free oracle ceiling (Reasoning + Perfect Perception).
- Metrics: The "perception share" is calculated as $(R_3 - R_4) / (R_1 - R_4)$ , quantifying how much of the total deficit is closed by providing perfect perception.

Key Results

Certified Ceiling: On a sample of 2,160 crystal structures, the phantom set was empty, confirming the ceiling is exactly 1.0000. Withholding cameras (reducing views) introduced phantoms and lowered the ceiling, proving the instrument measures the images rather than assuming them perfect.
Perception is Not the Bottleneck: For 13 out of 14 models, the post-perception residual (the gap between providing text geometry and the ceiling) was larger than the perception share. Supplying exact geometry as text improved all models, but closed less than half the gap for 13 of them.
No Correlation with Model Strength: The share of the deficit attributable to perception did not correlate with model strength. In fact, the strongest pixel performers were those for whom exact geometry added the least relative improvement, suggesting the bottleneck is reasoning, not vision, for top-tier models.
Vision-Only Superiority: A supervised vision model with no language component (ResNet-50) achieved 0.8952 accuracy on the same renders, outperforming every vision–language model. This demonstrates that the information required to solve the task is fully extractable from pixels without language or crystallographic priors.
Fabrication Detection: When a strong model was prompted to extract coordinates (the "extraction stage") without the reasoning question, it produced syntactically perfect lists that matched almost no real atoms (median recall 0.0000). A standard leaderboard would attribute this to bad reasoning, but the model-free reference identifies it as fabrication at the extraction stage.
Camera Placement vs. Count: The study found that camera placement (conditioning of the projection system) sets the noise budget for robustness, not the number of views. Four well-separated views were sufficient to empty the phantom set; adding a fifth view improved reconstruction fidelity but did not change identifiability.

Significance and Claims
The paper claims to provide the first model-free reference for separating perception and reasoning in multimodal evaluation. Its primary contributions are:

Attribution: It proves that for crystal structure tasks, the "headroom" for improvement lies primarily in the reasoning stage, not perception, for most current models.
Audit Primitive: It exposes a failure class (fabrication of intermediates) that downstream accuracy metrics miss. By providing a reference with no model inside it, the instrument can distinguish between a model that reasons poorly and one that hallucinates data during extraction.
Benchmark Design Rules: It offers concrete rules for benchmark builders: camera placement is more critical than view count for noise tolerance, and benchmarks should ensure the forward rendering is invertible to certify the ceiling.
Generalizability: The construction transfers to any benchmark where the forward rendering is known and invertible (e.g., molecular conformers, band structures), allowing for the separation of what a figure determines from what a model reads.

The authors maintain a modest scope, noting the method relies on known forward renderings and that the "post-perception residual" is an upper bound on reasoning rather than a direct measurement of it. However, they assert that the instrument fundamentally changes how multimodal deficits are interpreted, moving from model-vs-model comparisons to model-vs-geometry analysis.

Technical Summary: Separating Perception from Reasoning in Vision–Language Models via a Model-Free Render Ceiling

More like this