💬 NLP

BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams

This paper introduces BLUEX v2, a new benchmark comprising 395 open-ended questions from Brazil's leading university entrance exams (2022–2025) to evaluate 21 state-of-the-art LLMs on Portuguese discursive tasks, revealing significant performance gaps and highlighting challenges in mathematical reasoning and image understanding.

Original authors: João Guilherme Alves Santos, Giovana Kerche Bonás, Thiago Laitz, Thales Sales Almeida, Helio Pedrini

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: João Guilherme Alves Santos, Giovana Kerche Bonás, Thiago Laitz, Thales Sales Almeida, Helio Pedrini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've been testing how smart a group of robots are by asking them simple multiple-choice questions, like "What color is the sky? A) Blue, B) Red." They've been doing great at picking the right letter. But the researchers behind this paper, BLUEX v2, realized that picking a letter is easy. The real test of intelligence is writing a full essay, explaining why the sky is blue, and drawing a diagram to prove it.

This paper introduces a new, much harder "final exam" for AI models, specifically designed to test their ability to speak and write in Portuguese.

Here is the breakdown of their work using simple analogies:

1. The Problem: The "Multiple-Choice" Trap

Previously, researchers tested AI on Brazilian university entrance exams, but only the first round. Think of the first round as a trivia game where you just circle the right answer. It's easy for AI to guess or recognize patterns.

The second round of these exams (the one BLUEX v2 focuses on) is like a final thesis defense. Students must write long, detailed answers, solve complex math problems step-by-step, and interpret maps or graphs. The old tests couldn't measure if an AI could actually think and write like a human; they only measured if it could recognize the right answer.

2. The Solution: A New "Final Exam" for AI

The team created BLUEX v2, a massive dataset of 395 real questions from Brazil's top universities (UNICAMP and USP) between 2022 and 2025.

The Questions: These aren't just text. About 56% of them include images, charts, or maps that you must understand to answer correctly.
The Format: The AI has to write free-form answers, not pick A, B, C, or D.
The Subjects: It covers 9 different subjects, from History and Sociology to Physics and Math.

3. How They Graded the Robots: The "AI Teacher"

Grading a 5-page essay is hard for humans to do quickly for thousands of tests. So, the researchers built a system where an AI acts as the teacher.

The Rubric: First, they used an AI to read the "official correct answer" and break it down into a checklist (a rubric). For example, if the answer needs to mention "ATP" and "muscle contraction," the checklist has those two items.
The Judge: Another AI (the "Judge") reads the student robot's answer and checks off the items on the list. Did it mention ATP? Yes. Did it mention muscle contraction? No.
The Score: The robot gets a score from 0 to 10 based on how many checklist items it hit.
The Proof: To make sure the "AI Teacher" wasn't hallucinating, they compared its grading against real human teachers. The AI agreed with the humans 89.5% of the time. That's a very high score, proving the automated grading is reliable.

4. The Results: Who Passed and Who Failed?

They tested 21 different AI models (the "robots") on this exam.

The Spread: The scores ranged from a low of 4.18 to a high of 9.10. This shows the test is good at telling the difference between a smart robot and a not-so-smart one.
The Winners: The top performers were giant, expensive models like Gemini 3.1 Pro and GPT-5.
The Losers: Smaller, open-source models struggled, with the lowest scores around 4.18.
The Brazilian Stars: Interestingly, Brazilian-made models (Sabiá-4) performed very well, almost as good as the global giants, showing that local training helps.

5. The "Hard Parts": Where Robots Stumble

Even the best robots had trouble with two specific things:

Math Reasoning: This was the hardest subject. Just like a human who can write a great story but can't do long division, the AIs could write beautiful Portuguese essays but often messed up the math steps.
Image Understanding: When a question included a graph or a map, the robots' scores dropped by about 0.5 points on average. It's like giving a student a math problem but drawing the numbers on a crumpled piece of paper; the AI struggled to "see" the details clearly enough to solve the problem.

6. The Cost of the Test

The researchers were transparent about the price tag. Running this entire experiment—asking the questions, generating the image descriptions, and grading the answers—cost about $127. This is surprisingly cheap for a study of this size, showing that we can test AI rigorously without spending millions.

The Bottom Line

BLUEX v2 is a new, tougher standard for testing AI in Portuguese. It moves beyond simple trivia and forces the AI to write, reason, and look at pictures. The results show that while AI is getting very good at writing and arguing in Portuguese, it still struggles with complex math and interpreting visual data. This benchmark gives researchers a clear map of where to focus their next improvements.

Technical Summary: BLUEX v2

Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in various tasks, their evaluation in Portuguese remains underdeveloped, particularly regarding open-ended, discursive reasoning. Existing benchmarks, such as the original BLUEX, rely on multiple-choice questions from the first phase of Brazilian university entrance exams. These formats primarily test recognition and selection abilities, failing to capture the deeper understanding, multi-step reasoning, and complex articulation required in the second phase of these exams. Furthermore, current assessments often lack rigorous multimodal evaluation, despite the prevalence of images, graphs, and diagrams in academic testing. There is a critical need for a benchmark that evaluates Portuguese-speaking LLMs on free-form written responses across diverse academic subjects and cognitive capabilities.

Methodology

Dataset Construction (BLUEX v2)

The authors constructed a benchmark derived from the second-phase entrance examinations of Brazil's two leading universities: UNICAMP (Comvest) and USP (Fuvest), covering exam years 2022–2025.

Scope: The dataset contains 395 discursive questions (excluding essays/redações), which unfold into 919 graded subquestions.
Multimodality: 55.7% of questions include associated images (maps, graphs, diagrams) essential for the answer.
Annotation Pipeline: The construction involved a five-stage pipeline:
1. Data Collection: Extraction of questions and official expected answers from PDFs using a hybrid pipeline (pdfminer + OCR).
2. Data Cleaning: Human review by four annotators to correct OCR artifacts, verify segmentation, and remove duplicates/essays.
3. Captioning & Classification: Context-aware image captions were generated using Gemini 3.1 Flash Lite Preview (incorporating question text), and subject classification was performed using Sabiá-4.
4. Rubric Generation: For each subquestion, Sabiá-4 generated structured, machine-readable rubric criteria (binary checkable items) based on the official expected answers.
5. Taxonomy: Each question was annotated with six cognitive capability tags: Prior Knowledge (PRK), Text Understanding (TU), Image Understanding (IU), Mathematical Reasoning (MR), Brazilian Knowledge (BK), and Multilingual (ML).

Evaluation Protocol

Models: 21 state-of-the-art LLMs (including frontier API models and open-weight families) were evaluated.
Inference: Models were queried via API with a prompt containing the question, image captions, and contextual scaffolding (previous sub-questions and answers) to mirror the sequential nature of the exams.
LLM-as-a-Judge: Responses were graded by an LLM judge (Sabiá-4) against the generated rubric criteria. The judge outputs a binary verdict for each criterion, with the final score calculated as the ratio of met criteria ( $m/k$ ) scaled to a 0–10 range.
Validation: The LLM judge protocol was validated against two independent human reviewers on a sample of 200 criterion-level annotations. The LLM-human agreement (Cohen's $\kappa \approx 0.69$ ) fell within the "substantial" range, comparable to human-human agreement ( $\kappa \approx 0.82$ ).

Key Results

Overall Performance

Discriminative Power: The benchmark demonstrated a significant performance spread of 4.92 points (ranging from 4.18 to 9.10 on a 0–10 scale).
Top Performers: Frontier models dominated the top ranks, with Gemini 3.1 Pro Preview scoring 9.10, followed by GPT-5.4 (8.92) and GPT-5-mini (8.90).
Brazilian Models: Domestically developed models, Sabiá-4 and Sabiazinho-4, achieved competitive scores of 8.60 and 8.57, respectively.
Bottom Performers: Smaller open-weight models (e.g., LLaMA-3.2-11B Vision) scored significantly lower (4.18).

Capability and Subject Analysis

Hardest Dimensions: Mathematical Reasoning (MR) and Image Understanding (IU) emerged as the most challenging capabilities. MR had the highest cross-model standard deviation (1.92), making it the primary differentiator between models. IU averaged 7.79.
Subject Difficulty: STEM subjects (Mathematics, Physics, Chemistry) yielded lower average scores compared to Humanities (Philosophy, Sociology, History), reflecting the difficulty LLMs face with symbolic reasoning versus argumentative writing.
Multimodal Penalty: Questions containing images were, on average, 0.54 points harder than text-only questions across all models, indicating a persistent challenge in multimodal integration even for frontier models.

Significance and Contributions

The paper claims five primary contributions:

First Multimodal Open-Ended Benchmark: BLUEX v2 is the first benchmark derived from Brazilian second-phase university entrance exams, covering 9 subjects, 2 universities, and 4 years of data.
Rich Annotation: The dataset includes six cognitive capability tags and a four-stage construction pipeline, enabling fine-grained diagnostic analysis and full reproducibility.
Validated Evaluation Protocol: The authors propose a scalable LLM-as-a-judge protocol using generated rubrics, validated against human raters to achieve substantial agreement, facilitating annotation-free evaluation.
Comprehensive Evaluation: The study evaluates 21 models, revealing consistent failure modes in mathematical reasoning and image understanding across different model families.
Quantification of Multimodal Challenge: The work provides empirical evidence of the "multimodal penalty," showing that image-bearing questions are significantly harder in a real academic setting.

The authors conclude that while frontier models perform well on Portuguese argumentation, significant gaps remain in symbolic reasoning and visual interpretation. The benchmark serves as a tool to drive improvements in these specific areas and promotes transparency in Portuguese LLM evaluation. The dataset, code, and outputs are publicly available.