💻 computer science

Sci-Rho: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problems

This paper introduces Sci-Rho, a multilingual, visually-grounded symbolic benchmark comprising 4,242 executable STEM problem templates across five subjects and seven languages, which reveals significant robustness gaps in state-of-the-art Vision-Language Models when evaluated against worst-case variations rather than static averages.

Original authors: Muhammad Falensi Azmi, Ikhlasul Akmal Hanif, Vallerie Alexandra Putra, Adi Yeltay, Abdullah Mubarak, Fajri Koto

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammad Falensi Azmi, Ikhlasul Akmal Hanif, Vallerie Alexandra Putra, Adi Yeltay, Abdullah Mubarak, Fajri Koto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are testing a new student's ability to solve science problems. In the past, researchers would hand the student a single, static worksheet with one picture and one question. If the student got it right, they passed. But this paper argues that's like testing a driver only on an empty, straight road. It doesn't tell you if they can handle a sudden pothole, a detour, or a different weather condition.

The authors of this paper, Sci-ρ, decided to build a much tougher, more dynamic driving test for AI "students" (specifically, Vision-Language Models or VLMs).

Here is a simple breakdown of what they did and what they found:

1. The "Shape-Shifting" Test (The Dataset)

Instead of giving the AI 606 static science questions, the researchers built a digital factory.

The Blueprint: They created 606 "templates" for problems in Math, Physics, Chemistry, Biology, and Computer Science. Think of these as master blueprints.
The Factory: They wrote computer code (Python) that takes these blueprints and instantly prints out 10 slightly different versions of each problem.
- Example: If a math problem asks for the area of a square, the factory might print one version with a side length of 5, another with 7, another with a different color, or even a different shape entirely.
The Multilingual Twist: They didn't just do this in English. They translated the instructions into 7 languages (including Swahili, Hindi, and Arabic), while keeping the images the same.
The Result: They ended up with 42,420 unique problems. It's like giving the AI a massive library where every book has 10 different editions, all asking the same core logic question but looking slightly different.

2. The "Average" vs. "Worst-Case" Trap

The researchers tested 17 different AI models on this massive library. They looked at two scores:

Average Accuracy: How often the AI got the answer right across all 10 versions of a problem.
Worst-Case Accuracy: How often the AI got every single one of the 10 versions right.

The Big Discovery:
The AI models looked great on the "Average" score. But when the researchers looked at the "Worst-Case" score, the models fell apart.

The Analogy: Imagine a student who gets 9 out of 10 math problems right. They seem smart. But if you ask them to solve the same 10 problems again with slightly different numbers, and they get 4 of them wrong this time, they aren't actually "reasoning" through the math. They are likely just memorizing patterns or guessing based on how the numbers look.
The Finding: Even the smartest, most expensive AI models (like GPT-5.4) showed a huge gap. They could solve a problem easily, but if you changed the color of a graph or the numbers slightly, they often failed. Smaller, cheaper models failed even harder.

3. The Language Barrier

The researchers checked if the AI struggled more when the instructions were in a language other than English.

The Big Models: The giant, expensive models were like polyglots; they handled English, Chinese, and Swahili almost equally well.
The Small Models: The smaller, open-source models were like tourists who only speak English. When the instructions switched to a less common language (like Swahili), their performance dropped significantly. They seemed to get confused just by the language change, even though the science problem was the same.

4. The "Eyes vs. Brain" Problem

The researchers peeked inside one of the AI models to see how it "thought." They measured how much attention the model paid to the picture versus the text.

The Finding: The model's attention shifted depending on the language.
- When the text was in Chinese, the model relied less on the picture and more on the text.
- When the text was in Kazakh or Hindi, the model relied heavily on the picture.
The Metaphor: It's as if the AI thinks, "I don't understand this language well, so I'll just guess based on the drawing." When it understands the language perfectly, it trusts the words more. This suggests the AI isn't truly "seeing" the image in a human way; it's just using the image as a crutch when the text is confusing.

5. Where the AI Got Stuck

When the AI failed, the researchers categorized the errors:

Reading the Picture Wrong (63%): The AI couldn't count the dots in a diagram or read the numbers on a graph correctly.
Bad Logic (29%): The AI knew the right formula but applied it to the wrong thing, or made up facts that weren't there.
Math Errors (8%): The AI got the logic right but messed up the simple arithmetic.

The Bottom Line

The paper concludes that static tests are lying to us. Just because an AI can solve a specific science problem once doesn't mean it understands the science. It might just be recognizing a pattern it saw during training.

To truly know if an AI is smart, we need to shake the table, change the numbers, swap the languages, and see if it can still solve the problem. If it can't, it's not reasoning; it's just guessing. Sci-ρ is the tool designed to shake that table.

Technical Summary: Sci-ρ: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problems

Problem Statement

Current benchmarks for evaluating Large Vision-Language Models (VLMs) in STEM domains suffer from three primary limitations:

Lack of Robustness Evaluation: Most existing benchmarks rely on static, one-off images. They fail to distinguish between models that genuinely reason about visual content and those that succeed by matching surface features seen during training. A problem's visual presentation (e.g., specific coordinates or colors) is often incidental to its underlying logic, yet current evaluations do not test if models can solve logically equivalent variants with different visual parameters.
Narrow Domain Scope: Existing efforts are heavily concentrated on Mathematics and Physics, leaving Biology, Chemistry, and Computer Science largely unexplored in the context of visually-grounded reasoning, despite their distinct visual demands (e.g., molecular structures, biological diagrams).
Monolingual Bias: Evaluations are conducted almost exclusively in English, leaving it unclear whether VLM capabilities transfer to the multilingual settings where most learners and practitioners actually encounter STEM content.

Methodology: The Sci-ρ Benchmark

To address these gaps, the authors introduce Sci-ρ (Science Rhobustness), a dynamic, symbolic, and multilingual benchmark for visually-grounded STEM problems.

Dataset Construction

The dataset was constructed in two stages involving domain experts, including science Olympiad medalists and advanced undergraduates:

Template Creation: 606 high-quality problem templates were created across five subjects (Mathematics, Physics, Chemistry, Biology, and Computer Science) based on the Cambridge IGCSE and A-Level curricula. Each problem is implemented as an executable Python class. These classes dynamically generate problem instances by varying:
- Numerical values: Changing numbers in text or images.
- Visual patterns: Altering arrangements (e.g., object positions).
- Geometric shapes: Modifying dimensions or forms.
- Color schemes: Changing solution colors or object hues.
- Function types: Switching between linear, quadratic, etc.
  Each template guarantees at least 10 distinct variants.
Translation: The problem descriptions were translated into six additional languages (Indonesian, Chinese, Arabic, Hindi, Kazakh, and Swahili) using machine translation followed by rigorous verification by native speakers. Quality control included "honey-pot" errors to ensure translator accuracy.

The final dataset comprises 42,420 unique problem instances (606 templates × 10 variants × 7 languages), each paired with ground-truth solutions and step-by-step reasoning steps in LaTeX.

Evaluation Framework

The authors evaluated 17 state-of-the-art VLMs (including GPT-5.4, Gemini 2.5, Qwen3.5, Llama 4, InternVL, and others) using six metrics:

Average-case Accuracy ( $A_{avg}$ ): The mean accuracy across all generated variants.
Worst-case Accuracy ( $A_{wst}$ ): The proportion of problem templates where the model answered every variant correctly. This measures consistency.
Step-level Metrics: Recall ( $R$ ), Precision ( $P$ ), and F1 ( $F$ ) scores for reasoning steps, calculated by comparing model outputs against ground truth using an LLM judge (GPT-5.4).
Worst-case Step F1 ( $F_{wst}$ ): The minimum F1 score across all variants for a given template.
Image Attention Ratio (IAR): A mechanistic analysis measuring the relative attention VLMs allocate to image tokens versus text tokens during generation.

Key Results

1. The Robustness Gap

A significant performance gap exists between average-case and worst-case accuracy across all models.

Frontier Models: Even top-tier models like GPT-5.4 and Gemini 2.5 show drops of 10–28% in worst-case accuracy compared to their average accuracy. For instance, GPT-5.4's accuracy in Biology drops from 72.2% (average) to 47.8% (worst-case).
Smaller Models: Smaller open-weight models suffer severe degradation. In Biology, models like InternVL3.5-4B and Molmo2-4B effectively collapse under worst-case conditions, with accuracy dropping to near 0% for certain variants.
Architecture Sensitivity: The Qwen3.5-122B-A10B model (MoE architecture) showed significantly lower worst-case accuracy compared to its 27B dense counterpart, suggesting that parameter count alone does not guarantee robustness.

2. Multilingual Performance

Proprietary/Large Models: Large proprietary models (GPT-5.4, Gemini 2.5) and large open-weight models show near-zero performance degradation across languages, maintaining robustness in non-English settings.
Smaller Models: Smaller open-weight models exhibit notable performance drops in non-English languages, particularly in low-resource languages like Swahili. This suggests that pretraining corpus representation heavily influences multilingual robustness.

3. Reasoning Step Analysis

Step-level evaluation reveals that reasoning fragility is a general phenomenon, not limited to smaller models.

Consistency: There is a consistent gap between average F1 and worst-case F1 scores. Even GPT-5.4 drops 13–20 points in F1 score under worst-case conditions.
Subject Difficulty: Biology remains the most challenging subject across all models, followed by Mathematics and Computer Science, which show the largest drops in step-level consistency.

4. Mechanistic Insights (Attention Heads)

Analysis of the Qwen3-VL-8B-Instruct model's attention heads reveals substantial cross-lingual variation in the Image Attention Ratio (IAR):

Language Dependence: The model relies less on visual context when prompted in Chinese (lowest IAR) and significantly more when prompted in Kazakh and Hindi (highest IAR).
Hypothesis: The authors hypothesize that higher language familiarity allows the model to rely more on textual inputs, whereas lower familiarity forces a compensatory reliance on visual inputs.

5. Failure Modes

Manual inspection of GPT-5.4 failures identified three primary error types:

Figure-reading errors (63.5%): Failure to extract accurate numerical values or miscounting objects.
Knowledge errors (28.8%): Flawed logical reasoning, hallucinating non-existent information, or applying incorrect theorems.
Calculation errors (7.7%): Arithmetic mistakes despite correct logic.

Significance and Claims

The paper claims that Sci-ρ is the first symbolic, multilingual robustness benchmark for visually-grounded STEM reasoning. Its primary significance lies in shifting the evaluation paradigm from static, single-instance testing to dynamic, variant-based assessment.

The authors argue that:

Static benchmarks are insufficient: They fail to distinguish genuine reasoning from surface-level pattern matching. The large gap between average and worst-case accuracy exposes the fragility of current VLMs.
Robustness is a critical metric: True capability in STEM requires consistent performance across visual and numerical perturbations, a capability currently lacking even in frontier models.
Multilingual robustness is model-dependent: While large models generalize well across languages, smaller models struggle, highlighting the need for better multilingual pretraining or architectural adjustments.
Visual grounding is inconsistent: The variation in attention allocation across languages suggests that VLMs do not uniformly integrate visual and textual information, potentially relying on text when familiar and images when unfamiliar.

The work concludes that evaluating VLMs beyond static benchmarks is essential for measuring their true quality and readiness for rigorous STEM applications.