HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models

This paper introduces HSSBench, a comprehensive multilingual benchmark featuring over 13,000 samples generated through a novel expert-agent collaboration pipeline, designed to evaluate and address the current limitations of Multimodal Large Language Models in handling the interdisciplinary and abstract reasoning tasks characteristic of the Humanities and Social Sciences.

The Gist

Imagine you've built a brilliant robot student. You've tested it on math problems, science experiments, and coding challenges, and it's aced them all. It can solve complex equations faster than a calculator and write code without typos. You think, "This robot is a genius!"

But then, you hand it a picture of an old, dusty painting and ask, "What is this artist trying to tell us about the French Revolution?" Or you show it a graph of a company's profits and ask, "Why did this business fail, and what does it say about the economy of that time?"

Suddenly, the robot freezes. It might guess the wrong answer, make up facts, or stare blankly. It's like a brilliant chess player who suddenly can't understand a poem.

This is the problem HSSBench solves.

The Problem: The "STEM vs. Humanities" Gap

For a long time, we've been testing AI on STEM (Science, Technology, Engineering, Math). These subjects are like climbing a ladder: you go step-by-step, A leads to B, which leads to C, until you reach the single correct answer at the top. AI is great at climbing ladders.

But Humanities and Social Sciences (HSS)—like history, art, culture, and economics—are more like navigating a foggy forest. There isn't just one path. To understand a historical event or a piece of art, you need to connect dots across different fields, understand human emotions, cultural context, and "read between the lines." It requires horizontal thinking (connecting many different things) rather than vertical thinking (climbing one straight line).

Current AI models are like ladders in a forest; they get lost because they don't know how to wander and connect the dots.

The Solution: HSSBench (The "Forest Map")

The authors of this paper created a new test called HSSBench. Think of it as a massive, multi-language map of that foggy forest, designed specifically to see if AI can actually navigate it.

• The Size: It's huge. It contains over 13,000 questions covering six major areas: Geography, Art, Culture, Social Sciences, History, and Economics.
• The Languages: It's not just in English. It's in the six official languages of the United Nations (English, Chinese, French, Russian, Spanish, and Arabic), because culture and history look different depending on where you are in the world.
• The Format: Instead of just text, it uses images. You might see a photo of a rock formation and have to guess its geological age, or look at a painting and guess the cultural significance of the costumes.

How They Built It: The "Expert Kitchen"

You can't just ask a computer to write these questions; it would be like asking a robot to write a novel about human heartbreak without ever feeling love. The data would be shallow and full of errors.

So, the team built a special "kitchen" to cook up the data:

1. The Chefs (Human Experts): Real professors and specialists in history, art, and economics gathered raw ingredients (old photos, textbooks, maps).
2. The Sous-Chefs (AI Agents): They used AI to help chop the ingredients, organize the recipes, and generate draft questions.
3. The Tasting Committee: Both the human experts and the AI checked every single question to make sure it wasn't a trick question, that the answer was correct, and that it actually required looking at the picture to solve.

This ensured the test was high-quality and fair.

The Results: The Robot is Still a Toddler in the Forest

When they ran the test on over 20 of the smartest AI models in the world (including giants like GPT-4 and Qwen), the results were surprising: The robots struggled.

• The Score: Even the best models only got about 40-50% of the answers right. That's barely passing a high school exam!
• The "Human" Score: Real human experts got over 93% right.
• The Hardest Part: The models did especially poorly on Economics and Open-ended questions (where you have to explain why without multiple-choice hints).
• The "Hallucination" Issue: When the models tried to "think step-by-step" (a technique called Chain-of-Thought), they often got worse. It's like when you try to explain a joke to someone, and in trying to explain the logic, you ruin the punchline. The AI got confused by its own reasoning.

Why This Matters

This paper is a wake-up call. It tells us that just because an AI can solve a math problem, it doesn't mean it "understands" the world.

• Real-world impact: If we want AI to help doctors, lawyers, historians, or policymakers, it needs to understand human culture and context, not just numbers.
• The Future: HSSBench gives researchers a target. Now that we have a map of where the AI is failing, we can start building better models that can truly "think" like humans, connecting art, history, and economics into a coherent picture of our world.

In short: HSSBench is the test that finally asks, "Can you understand the human story?" and currently, the answer from our smartest robots is a hesitant, "Not really, not yet."

Technical Summary

Here is a detailed technical summary of the paper "HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models."

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have achieved significant success in STEM (Science, Technology, Engineering, Mathematics) domains, there is a critical gap in evaluating their capabilities in the Humanities and Social Sciences (HSS).

• Reasoning Mismatch: STEM tasks typically rely on "vertical reasoning" (sequential logical deduction to find a single correct answer). In contrast, HSS tasks require "horizontal reasoning," involving interdisciplinary connections, cultural context, and the generation of multiple valid interpretations.
• Cross-Modal Failure: Current MLLMs struggle to link abstract HSS concepts with their visual representations. As illustrated in the paper's Figure 2, models may know the definition of a term (e.g., "Business Penmanship") but fail to recognize it when presented in an image, indicating a failure in cross-modal knowledge transfer.
• Lack of Benchmarks: Existing benchmarks are either too general, lack depth in HSS, or rely heavily on web-crawled data without rigorous expert validation, leading to potential biases and insufficient evaluation of complex cultural and historical reasoning.

2. Methodology

The authors propose HSSBench, a large-scale, multilingual benchmark designed specifically to assess MLLMs on HSS tasks. The methodology involves a novel data generation pipeline and rigorous evaluation protocols.

A. Dataset Construction Pipeline (VGP)

The dataset was built using a Visual Question Generation Pipeline (VGP) involving three stages:

1. Dataset Preparation (Stage I):
   • Data Collection: A hybrid approach combining human experts (providing private, licensed, or drawn images/text) and a Networked Information-Aggregation Agent (scraping and filtering web data based on keywords).
   • Sources: Materials were sourced from the six official UN languages (English, Chinese, French, Russian, Spanish, Arabic) to ensure linguistic and cultural diversity.
2. Dataset Construction (Stage II):
   • Expert Construction: Experts revised existing questions and created new ones, ensuring the connection between images and questions was non-trivial (i.e., the answer cannot be derived from text alone or image alone).
   • Multi-Agent Construction: An automated framework utilizing a Summarizer, Extractor, Question Generator, and Image Matcher to scale data production. The generator uses Chain-of-Thought (CoT) prompting to create high-quality multiple-choice questions (MCQs).
3. Validation (Stage III):
   • Agent Validation: Filters out redundant questions and verifies that a question requires both modalities (image and text) to be answered correctly.
   • Expert Validation: Domain experts verify accuracy, lack of ambiguity, and absence of sensitive/cultural bias.

B. Dataset Statistics

• Scale: 13,152 meticulously curated samples.
• Categories: 6 major categories (Geography, Art, Culture, Social Sciences, History, Economy) covering 45 specialized subtypes.
• Format: Primarily multiple-choice questions (reformulated from multi-answer sources for consistency), translated and validated across six languages.

C. Experimental Setup

• Models Evaluated: Over 20 MLLMs, including open-source models (Qwen2.5-VL series, InternVL, LLaVA, etc.) and closed-source models (GPT-4o, GPT-4.1 series).
• Evaluation Strategies:
  • Prompting: Direct Answer vs. Chain-of-Thought (CoT).
  • Question Type: Multiple-Choice vs. Open-Ended.
  • Robustness Checks: Experiments with "confounding options" (e.g., "None of the above") and Retrieval-Augmented Generation (RAG).
  • Visual Dependency: Replacing images with AI-generated text descriptions vs. expert annotations to test visual information extraction.

3. Key Contributions

1. HSSBench Dataset: The first large-scale, multilingual benchmark dedicated to HSS multimodal reasoning, covering 6 categories and 45 subtypes with 13k+ samples.
2. Novel Data Pipeline: A hybrid human-agent framework that ensures high-quality, culturally diverse, and multimodal-dependent data generation, addressing the scarcity of expert-annotated HSS datasets.
3. Comprehensive Benchmarking: Extensive evaluation of 20+ SOTA models across six languages, revealing specific weaknesses in HSS reasoning compared to STEM.
4. Insightful Analysis: Identification of specific failure modes, such as the inability to integrate visual cues with abstract concepts and the degradation of performance when answer choices are removed.

4. Key Results

• Overall Performance: HSSBench poses a significant challenge. Even state-of-the-art models (e.g., GPT-4.1, Qwen2.5-VL-72B) achieve accuracies generally below 60%, far below human expert performance (~94%).
• Category Difficulty:
  • Most Difficult: Economy tasks (lowest average scores), requiring deep theoretical understanding and complex reasoning.
  • Easiest: Geography tasks (highest scores), likely due to the factual and spatial nature of the data which aligns better with current model training.
• Prompting & Format:
  • CoT Limitations: CoT prompting does not consistently help; in some cases, it exacerbates hallucinations and leads to incorrect reasoning paths.
  • Open-Ended vs. MC: Performance drops drastically in open-ended settings (often <15% accuracy), suggesting models rely heavily on the "answer space" provided by options rather than genuine reasoning.
• Multilingual Performance: Models generally perform better in Chinese and English, with lower scores in other UN languages, highlighting a language bias.
• RAG Ineffectiveness: Standard Retrieval-Augmented Generation (using Wikipedia/general docs) failed to consistently improve performance, indicating that HSS tasks require specialized, domain-specific retrieval and integration mechanisms.
• Visual Dependency: When images were replaced by text descriptions, performance dropped significantly, confirming that current models struggle to extract critical visual details necessary for HSS reasoning.

5. Significance

• Paradigm Shift: The paper argues that true intelligence requires not just STEM logical deduction but also the ability to interpret cultural, historical, and social contexts visually.
• Research Direction: HSSBench establishes a new standard for evaluating "horizontal reasoning" and cross-modal knowledge transfer, pushing the community to move beyond simple factual recall.
• Future Work: The results suggest that future MLLM development must focus on:
  • Internalizing complex, interdisciplinary knowledge.
  • Improving visual grounding for abstract concepts.
  • Developing specialized RAG systems for HSS domains.
  • Addressing cultural and linguistic biases in training data.

In conclusion, HSSBench reveals that despite rapid progress in general multimodal capabilities, MLLMs remain fundamentally limited in the Humanities and Social Sciences, particularly in tasks requiring deep cultural context and the synthesis of visual and abstract information.