💬 NLP

PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understanding

This paper introduces PUMA, a novel 900-task benchmark designed to evaluate the capabilities of multimodal AI models in the Polish cultural and linguistic context, revealing significant performance gaps in audio and document understanding despite strong visual question-answering results.

Original authors: Sławomir Dadas, Michał Perełkiewicz, Rafał Poświata, Małgorzata Grębowiec, Bartłomiej Jaworski, Izabela Woźniakowska

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sławomir Dadas, Michał Perełkiewicz, Rafał Poświata, Małgorzata Grębowiec, Bartłomiej Jaworski, Izabela Woźniakowska

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving field of artificial intelligence, computers are learning to see, hear, and read just as humans do. For years, these systems were trained primarily on text, learning to process words and sentences. Now, they are expanding their senses to include images, audio recordings, and complex documents. However, a significant gap remains in how well these machines understand the world beyond English. Just as a person can speak a language fluently yet miss the subtle jokes, historical references, or cultural traditions embedded within it, an artificial intelligence can process words correctly while failing to grasp their deeper meaning. Understanding a culture requires more than grammar; it demands familiarity with local history, geography, popular trends, and the unspoken rules of daily life. Without this cultural grounding, even the most advanced systems risk offering superficial or incorrect interpretations of the world they are trying to understand.

To address this blind spot, researchers at the National Information Processing Institute in Warsaw, Poland, have created a new testing ground called PUMA. This benchmark is not a single test but a collection of 900 carefully crafted challenges designed specifically to probe the limits of artificial intelligence within the Polish cultural and linguistic context. The team built these tasks by hand, drawing from real-world examples of Polish history, modern life, geography, and audio recordings. The goal was to see if machines could truly understand Poland, not just translate its words. The benchmark covers three distinct ways humans perceive the world: images, sound, and documents. In the image section, models are asked to identify historical landmarks, recognize contemporary pop culture figures, or distinguish between different types of Polish landscapes. The audio section tests whether a machine can transcribe speech from noisy recordings, understand the intent behind a spoken conversation, or even identify non-speech sounds like traditional folk instruments or specific environmental noises. The document section challenges the systems to read handwritten notes, extract data from complex tables, and interpret visually rich pages like menus or official forms.

When the researchers put dozens of the world's most advanced artificial intelligence systems through this rigorous test, the results revealed a stark divide between general capability and cultural fluency. The top-performing commercial models, particularly those from Google's Gemini family, achieved high scores, demonstrating a strong ability to handle visual questions and document analysis. These systems managed to score nearly 80 percent on the most difficult tasks, showing they can effectively navigate the visual and textual landscape of Poland. However, the study found that even these leading models struggle significantly when the task moves away from clear text and images into the realm of sound. Non-speech audio, such as identifying specific musical instruments or environmental sounds unique to the region, proved to be the most difficult category, with the best models scoring only around 56 percent. This suggests that while machines are becoming excellent at reading and seeing, their ability to "hear" and interpret the nuanced sounds of a culture remains underdeveloped.

The evaluation also highlighted a surprising weakness in how some systems handle the practical side of language. When asked to extract specific data from a document and format it perfectly, many models failed, often tripping over minor structural errors. Furthermore, the study noted that some of the most famous American models refused to answer questions about Polish historical figures or cultural works, treating them as sensitive topics when they were not. This refusal rate was much higher than in other model families, effectively lowering their scores on knowledge-based questions. In contrast, specialized models designed for specific tasks, such as converting speech to text or reading printed characters, often outperformed the massive, general-purpose systems in those narrow areas. This indicates that for specific, practical applications, smaller, focused tools may still be more reliable than the all-encompassing giants.

Ultimately, the PUMA benchmark serves as a mirror, reflecting the current state of artificial intelligence in a specific cultural context. It shows that while machines are making impressive strides in visual and textual understanding, they still lack the deep, intuitive grasp of local culture that comes from lived experience. The researchers found that progress in this field cannot be measured solely by how well a system performs on English-language tests or general knowledge quizzes. True understanding requires the ability to navigate the messy, rich, and often noisy reality of a specific culture, from the sound of a local dialect to the layout of a handwritten note. By opening their tools and data to the public, the team hopes to encourage the development of artificial intelligence that is not just globally capable, but locally grounded, ensuring that these powerful systems can serve and understand people in their own languages and cultural settings.

Technical Summary: PUMA (Polish Unified Multimodal Assessment)

Problem Statement

While Large Language Models (LLMs) have rapidly evolved to support multimodal inputs (images, audio, documents), their evaluation has largely remained centered on English and Western cultural contexts. Existing benchmarks often fail to assess whether models can effectively combine linguistic information with cultural nuances, historical references, and local practical skills in non-English languages. Specifically, there is a lack of comprehensive frameworks that jointly evaluate image, audio, and document understanding within a single cultural context, particularly for Polish. Current evaluations often rely on text-only tasks or general knowledge that does not require deep cultural grounding, leading to a gap in understanding how models perform on culturally specific, multimodal tasks.

Methodology

The authors introduce PUMA (Polish Unified Multimodal Assessment), a benchmark consisting of 900 hand-crafted tasks designed to probe the limits of multimodal models in the Polish linguistic and cultural context.

Benchmark Structure

The dataset is organized into three modalities, each containing three distinct categories (100 tasks per category):

Images:
- History and Culture: Tests knowledge of Polish history, traditions, art, architecture, and significant figures.
- Contemporary Life: Covers modern pop culture, media, politics, sports, and daily life.
- Geography & Environment: Focuses on Polish geography, natural landmarks, flora, fauna, and infrastructure.
Audio:
- Automatic Speech Recognition (ASR): Evaluates transcription capabilities on challenging, noisy, and dialectal recordings.
- Speech QA: Assesses comprehension of spoken Polish, requiring reasoning about speaker intent and cultural context.
- Sound & Music QA: Tests recognition of non-speech audio (environmental sounds, instruments, jingles) tied to Polish culture.
Documents:
- Optical Character Recognition (OCR): Evaluates text extraction from visually challenging documents (handwriting, low-quality scans, tables).
- Document QA: Tests understanding of visually rich documents (charts, infographics, structured text).
- Structured Extraction: Requires extracting specific data from documents and outputting it in a predefined JSON format.

Evaluation Framework

PUMA employs a deterministic, rule-based evaluation approach to ensure reproducibility and cost-efficiency, avoiding the need for external human judges or LLM-as-a-judge systems.

Scoring: Uses a binary "strict score" (0 or 1) based on predefined verification rules. A "soft score" (fractional) is also calculated to measure partial correctness.
Verification Rules: Includes seven types of rules:
- For QA: Include, Exclude, Regex, and Order (using normalized Kendall Tau distance). Text normalization (lemmatization, lowercasing) is applied to handle Polish morphology.
- For ASR: Uses Word Accuracy (WAcc), defined as $1 - \text{WER}$ , with a strict threshold of 0.9.
- For OCR: Combines WAcc for text and the Tree-Edit-Distance-based Similarity (TEDS) metric for table structure.
- For Structured Extraction: Compares JSON objects against a reference schema, calculating similarity based on matching field values.

Experimental Setup

The authors evaluated over a dozen open-weight and commercial multimodal models, as well as over 50 Visual Language Models (VLMs) for visual tasks. Experiments were conducted with temperature set to 0 (with iterative increases to 0.3 to prevent repetition loops in reasoning models) and maximum token budgets for image representation.

Key Results

The evaluation highlights a significant performance gap between frontier commercial models and other systems, as well as specific weaknesses in non-text modalities.

Top Performers: The Gemini family of models (particularly Gemini-3.1-Pro and Gemini-3.5-Flash) achieved the highest scores, with Gemini-3.1-Pro reaching nearly 80% average accuracy across all categories. Claude-Fable-5 performed exceptionally well in document-based categories but lacks audio support.
Open-Weight Models: Open models generally lagged behind commercial solutions. Phi-4-Multimodal-Instruct struggled significantly with Polish grammar and task difficulty. Mistral models showed slight improvements but remained far behind commercial leaders. However, smaller specialized models like Gemma-4 and Qwen3.5 showed strong performance in document-related categories, matching some commercial models.
Modality Challenges:
- Audio: Sound & Music QA was the most difficult category, with the best model achieving only 56%. This suggests a lack of training data for non-speech audio.
- Structured Extraction: This task proved challenging for many models; even minor errors in JSON structure resulted in a failed response, leading to high variance in scores.
- ASR vs. OCR: Specialized ASR models (e.g., ElevenLabs, Whisper-Large-V3) performed competitively with or better than general multimodal models. In OCR, Claude-Fable-5 and Gemini-3.5-Flash led, while dedicated OCR models often underperformed under strict metrics compared to their soft scores.
Refusals: OpenAI models (GPT series) exhibited higher refusal rates for questions involving historical figures or art, which negatively impacted their scores.

Significance and Contributions

The paper claims the following contributions and significance:

First Polish Multimodal Benchmark: PUMA is the first benchmark to jointly evaluate image, audio, and document understanding within a single framework for the Polish language, combining culturally grounded tasks with practical skills (ASR, OCR, structured extraction).
Cultural Grounding: It demonstrates that mastery of grammar and vocabulary is insufficient for full language understanding; models require deep familiarity with local history, traditions, and social contexts to avoid superficial or incorrect interpretations.
Evaluation Framework: The authors provide an open-source tool for building multimodal question datasets and running deterministic evaluations, advancing localized multimodal AI research.
Identifying Gaps: The results reveal that while frontier models excel in visual question answering, significant gaps remain in audio understanding (especially non-speech) and complex document processing. The paper argues that progress in multimodal AI cannot be reliably assessed using English-centric or text-only benchmarks alone.

Limitations

The authors acknowledge several limitations:

Rule-Based Constraints: The deterministic approach limits the inclusion of highly open-ended tasks.
Modality Coverage: Video is not included due to limited current support in models, though the framework supports it.
Temporal Scope: The dataset focuses on stable, well-known facts and avoids rapidly changing trends to ensure long-term stability.
Cultural Diversity: While curated for diversity, the annotator group was relatively homogeneous in age and experience, potentially limiting the breadth of generational perspectives.
Data Release: To prevent data leakage and maintain benchmark integrity, only a subset of the dataset is publicly released, with a public leaderboard for evaluation results.