💬 NLP

MARCA: A Checklist-Based Benchmark for Multilingual Web Search

The paper introduces MARCA, a bilingual (English and Portuguese) benchmark featuring 52 manually curated questions with checklist-style rubrics to evaluate and compare the web-search capabilities, orchestration strategies, and multilingual performance of 14 large language models.

Original authors: Thales Sales Almeida, Giovana Kerche Bonás, Ramon Pires, Celio Larcher, Hugo Abonizio, Marcos Piau, Roseval Malaquias Junior, Rodrigo Nogueira, Thiago Laitz

Published 2026-04-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thales Sales Almeida, Giovana Kerche Bonás, Ramon Pires, Celio Larcher, Hugo Abonizio, Marcos Piau, Roseval Malaquias Junior, Rodrigo Nogueira, Thiago Laitz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of super-smart robots (Large Language Models, or LLMs) that can answer any question you ask. Usually, you just ask them, and they pull answers from their memory. But what if you need them to find brand new information on the internet, like "Who won the 82nd Golden Globes?" or "List all the current governors of Brazil"?

The problem is that these robots sometimes get lazy, make things up (hallucinate), or miss important details. Also, most tests only check if they are good at finding things in English. But what about Portuguese?

This paper introduces MARCA, a new "report card" designed to test how well these robots can search the web, read the results, and put together a perfect answer in both English and Portuguese.

Here is a simple breakdown of how they did it and what they found, using some everyday analogies:

1. The Test: The "Checklist" Game

Instead of just asking the robot, "Is the answer right?", the researchers gave the robots a very specific Checklist.

The Analogy: Imagine you hire a personal assistant to plan a dinner party. You don't just say, "Make it good." You give them a checklist:
1. Buy 5 steaks.
2. Get a bottle of red wine.
3. Invite 4 friends.
4. Turn on the oven.
The MARCA Twist: The researchers created 52 complex questions (like "List all the winners of the Golden Globes film categories"). For each question, they wrote a checklist of exactly what must be in the answer.
The Goal: If the robot forgets even one winner on the list, it gets a lower score. This stops the robot from giving a "vague but mostly correct" answer. It forces them to be precise.

2. The Two Ways to Work: "The Solo Artist" vs. "The Conductor"

The researchers tested the robots in two different ways to see how they handle big tasks.

Mode A: The Solo Artist (Basic Framework)
- How it works: One robot tries to do everything alone. It searches, reads websites, and writes the answer all by itself.
- The Metaphor: It's like one person trying to juggle 10 balls at once. They might drop a few.
Mode B: The Conductor (Orchestrator Framework)
- How it works: One "Conductor" robot breaks the big question into smaller pieces and hires other "Sub-agent" robots to do the specific work. The Conductor doesn't search the web; it just manages the team.
- The Metaphor: It's like a movie director. The director doesn't act, paint, or sew the costumes. They tell the actors, the painters, and the seamstresses what to do, then put it all together.
- The Result: For most robots, being a "Conductor" worked much better. Breaking a big, messy task into small, focused jobs helped them get more items on the checklist correct.

3. The Language Gap: English vs. Portuguese

The researchers wanted to see if the robots were equally good at searching in English and Portuguese.

The Finding: It's a mixed bag.
- Some robots are like polyglots: They perform just as well in Portuguese as they do in English.
- Other robots are like tourists who only speak English: When asked to search in Portuguese, they struggle. They might try to translate the question into English in their head, search for English results, and then translate them back. This works fine for general topics, but fails when the answer only exists in Portuguese (like local Brazilian government details).
The Takeaway: Just because a robot is smart in English doesn't mean it's smart in Portuguese. The "internet" looks different in different languages, and the robots need to know how to navigate those specific streets.

4. The Price Tag: Better Answers Cost More

Finally, they looked at the cost.

The Analogy: Think of it like ordering food.
- Cheap Meal: A quick, simple answer (Low cost, low accuracy).
- Gourmet Meal: A detailed, perfectly researched answer with a full checklist (High cost, high accuracy).
The Result: Generally, if you want the robot to get a near-perfect score (90%+ on the checklist), you have to pay more. However, some robots are "efficient chefs"—they can make a great meal without costing a fortune, while others are expensive but still make mistakes.

Summary

MARCA is a new tool that stops us from just trusting robots blindly. It forces them to prove they did their homework by checking off specific items on a list.

Key Lesson 1: Giving robots a team (Orchestrator) usually helps them get better answers.
Key Lesson 2: Being good at English doesn't guarantee being good at Portuguese.
Key Lesson 3: If you need a perfect answer, you usually have to pay for it, but some robots offer better value than others.

This research helps developers choose the right robot for the job, especially if they are building tools for Portuguese speakers who need reliable, fact-checked information from the web.

1. Problem Statement

Large Language Models (LLMs) are increasingly relied upon as information sources, yet their reliability hinges on the ability to search the web, select relevant evidence, and synthesize complete answers without hallucinations. While recent benchmarks (e.g., BrowseComp, WebArena) evaluate web-browsing and agentic capabilities, they suffer from two critical limitations:

English-Centricity: Most evaluations focus exclusively on English, ignoring the performance dynamics of other major languages.
Lack of Granularity: Existing metrics often measure task completion rather than the correctness and completeness of the retrieved information, particularly for complex queries involving multiple entities.

Portuguese, spoken by over 250 million people, is notably underrepresented in web-search benchmarks. There is a lack of understanding regarding how models transfer from English to Portuguese, how they handle language-specific retrieval dynamics, and whether they can effectively synthesize evidence from non-English sources.

2. Methodology: The MARCA Benchmark

The authors introduce MARCA (Maritaca AI Research Checklist evAluation), a bilingual (English and Portuguese) benchmark designed to evaluate LLMs on web-based information seeking.

2.1 Dataset Construction

Scale: 52 manually authored questions distributed across 9 thematic domains.
Task Type: Multi-entity information seeking. Questions require the model to identify, retrieve, and associate attributes for multiple distinct entities (e.g., "List winners for all Film categories at the 82nd Golden Globes").
Evaluation Rubrics: Each question is paired with a manually curated checklist.
- The checklist specifies expected entities and required attributes.
- Evaluation is binary (satisfied/not satisfied) for each item.
- Dynamic Checklists: Some questions use dynamic rubrics where expected values (e.g., citation counts) are scraped at evaluation time to ensure the benchmark reflects the current state of the web rather than a static snapshot.
Bilingual Parity: The dataset includes parallel English and Portuguese versions with corresponding checklists to isolate language as a variable.

2.2 Evaluation Frameworks

The paper evaluates models under two distinct interaction architectures to study the impact of task decomposition:

Basic Framework: A single model instance has direct access to web_search and web_scrape tools. It must manage the entire information-seeking loop (querying, scraping, synthesizing) within a single context window (max 40 iterations).
Orchestrator Framework: A hierarchical multi-agent system.
- A top-level Orchestrator decomposes the complex query into sub-questions.
- Subagents (delegated by the orchestrator) perform the actual web interactions (search/scrape).
- The Orchestrator synthesizes the partial results from subagents into a final answer.
- Constraint: The Orchestrator does not interact with the web directly; it relies on subagent outputs.

2.3 Evaluation Protocol

Stochasticity Control: Each question is executed 3 times ( $k=3$ ) to account for model randomness.
Scoring: An LLM judge (GPT-4.1) grades the model's output against the checklist, producing a Checklist Accuracy score:
$\text{Checklist Accuracy} = \frac{C}{T}$
Where $C$ is the number of satisfied requirements and $T$ is the total requirements.
Uncertainty Quantification: Results are reported as Mean ( $\mu$ ) $\pm$ Standard Deviation ( $\sigma$ ) across the $k$ runs to capture run-level variability.

3. Key Contributions

First Bilingual Web-Search Benchmark for Portuguese: Provides a parallel English/Portuguese dataset specifically designed to evaluate evidence-grounded answering and multi-entity retrieval.
Checklist-Based Rubrics: Moves beyond simple pass/fail metrics to fine-grained evaluation of completeness and correctness using human-verified checklists.
Framework Comparison: Systematically compares "Flat" (Basic) vs. "Hierarchical" (Orchestrator) agent architectures, offering insights into how task decomposition affects performance.
Multilingual Analysis: Offers empirical evidence on how LLMs degrade (or adapt) when moving from English to Portuguese in a web-search context.

4. Key Results

The authors evaluated 14 models (including GPT-4/5, Gemini, Qwen, and Maritaca's Sabia models).

4.1 Performance Variance

Top Performers: Models like Gemini-3-pro and GPT-5.2 achieved high checklist coverage (~0.90) in optimal configurations.
Bottom Performers: Weaker models (e.g., Gemini-2.5-Flash-Lite) scored as low as 0.26, highlighting significant gaps in the ability to enumerate and attribute multiple entities reliably.
Stability: High-performing models generally showed low run-level variance ( $\sigma$ ), while some models (e.g., Gemini-2.5-Pro) exhibited high sensitivity to stochasticity.

4.2 Impact of Framework (Basic vs. Orchestrator)

General Trend: The Orchestrator framework generally improved Checklist Accuracy over the Basic framework.
Mechanism: Task decomposition helps models struggling with context management by allowing subagents to focus on specific entities.
Exceptions: Some strong models (e.g., GPT-4.1) performed slightly better or comparably in the Basic setting, suggesting that for highly capable models, the overhead of coordination and potential context loss in the Orchestrator setup can be detrimental.

4.3 Impact of Language (English vs. Portuguese)

Heterogeneous Transfer: Performance transfer from English to Portuguese is inconsistent across model families.
- Robust Models: Some (e.g., GPT-4.1, Sabiazinho-4) maintained or slightly improved performance in Portuguese.
- Degraded Models: Others (e.g., Qwen-3-235B, Gemini-2.5-Pro) showed significant drops (>5% accuracy) in Portuguese.
Root Cause: The degradation is not just due to language understanding but also retrieval behavior. Models may implicitly translate queries to English, failing to access Portuguese-specific content (e.g., local government data or regional sports details) that is not present in English sources.

4.4 Performance vs. Cost

There is a general positive correlation between inference cost (token usage) and Checklist Accuracy.
However, the relationship is not linear; some models offer significantly better accuracy-per-cost ratios, providing actionable guidance for practitioners to select models based on budget constraints.

5. Significance and Conclusion

MARCA addresses a critical gap in the evaluation of agentic LLMs by providing a rigorous, multilingual standard for web search. The study reveals that:

Multilingual capability is not uniform: High performance in English does not guarantee proficiency in Portuguese web search, particularly regarding evidence consolidation.
Architecture matters: While orchestration (delegation) generally aids complex multi-entity tasks, it is not a universal fix and depends on the base model's reasoning capabilities.
Evaluation needs granularity: Checklist-based rubrics are essential for diagnosing specific failures (e.g., missing entities vs. hallucinated attributes) rather than just measuring task completion.

The benchmark is publicly available, encouraging further research into improving the reliability and multilingual robustness of web-searching AI agents.