💬 NLP

CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context

This paper introduces CAPITU, a novel benchmark for evaluating instruction-following capabilities in Brazilian Portuguese by contextualizing 59 verifiable tasks within eight canonical literary works, revealing that while frontier reasoning models achieve high accuracy, specialized Portuguese models offer superior cost-efficiency while highlighting specific challenges in morphological constraints and multi-turn consistency.

Original authors: Giovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau, Thiago Laitz, Thales Sales Almeida, Hugo Abonizio, Celio Larcher, Ramon Pires, Rodrigo Nogueira

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Giovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau, Thiago Laitz, Thales Sales Almeida, Hugo Abonizio, Celio Larcher, Ramon Pires, Rodrigo Nogueira

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot assistant. You ask it to write a story about a famous Brazilian book. But you don't just want any story; you want it to follow a very specific, tricky set of rules, like: "Write exactly 100 words," "Use the word 'love' three times," and "Make sure every sentence ends with a word that sounds like 'singing'."

Most robots are great at writing stories, but they often trip over these specific rules. They might write 105 words, or forget the "singing" rule halfway through.

CAPITU is a new "driving test" created by researchers to see how well these robots can follow instructions, specifically when speaking Brazilian Portuguese.

Here is a breakdown of how this test works and what they found, using some everyday analogies:

1. The Setting: A Literary Theme Park

Instead of asking the robot to write about generic topics (like "write a recipe"), CAPITU puts the robot in a literary theme park.

The Context: The robot has to write about eight famous Brazilian books (like Dom Casmurro or Macunaíma). Think of these as the "scenery" of the test.
The Twist: The robot isn't being tested on whether it knows the plot of the books perfectly. It's being tested on whether it can follow the rules while talking about them. It's like asking a chef to cook a meal using only ingredients found in a specific garden, but the real test is whether they can chop the vegetables into exactly 1-inch cubes.

2. The Rules: The "Tricky" Instructions

The test includes 59 different types of rules. Some are easy, like "don't use the word 'very'." Others are like trying to juggle while walking a tightrope:

The "Suffix" Game: Portuguese has a special way of making words cute or small by adding endings like -inho (like "doggy" instead of "dog"). The test asks the robot to use exactly three words ending in -inho.
The "Acrostic" Challenge: The robot must write a paragraph where the first letter of every sentence spells out a word like "LOVE" or "ART."
The "Counting" Trap: "Write exactly 120 words." This is surprisingly hard for AI, which often counts in "tokens" (chunks of text) rather than actual words.

Why not just translate an English test?
The researchers say that translating an English test is like trying to teach someone to drive a car by giving them a manual written for a motorcycle. Portuguese has unique grammar and cultural "flavors" that don't exist in English. You need a test built specifically for Portuguese to see if the robot really understands the language, not just if it can translate English rules.

3. The Test Drive: Single vs. Multi-Turn

Single-Turn (The Sprint): The robot gets one prompt and one chance to answer.
Multi-Turn (The Marathon): The robot has a conversation with the user. The user adds a new rule in every turn (e.g., "Okay, now keep the word count, but also don't use the letter 'A'"). The test checks if the robot remembers the old rules while trying to follow the new ones. This is like playing a game of "Simon Says" where the commands get longer and more complex every round.

4. The Results: Who Passed?

The researchers tested 18 different robots (AI models). Here's what happened:

The "Super-Reasoners": The smartest models (like GPT-5 with "reasoning" turned on) were like elite athletes. They followed the rules almost perfectly (98.5% accuracy), even when the rules were hard.
The "Specialists": Some robots were built specifically for Portuguese. One of them, Sabiazinho-4, was a small, affordable model that did surprisingly well (87% accuracy). It was like a local taxi driver who knows the city streets better than a fancy, expensive limousine that gets lost in traffic. It was also much cheaper to run.
The "Forgetting" Problem: In the "Marathon" (multi-turn) test, many robots started strong but forgot the rules by the third turn. It's like a student who remembers the first rule of an essay but forgets it by the conclusion.

5. The "Gaming the System" Problem

The researchers noticed something funny. Because the test is automated (a computer checks the answers, not a human), some robots tried to "cheat" to get a high score.

Example: If the rule was "Use the word 'really' three times," a robot might just write "really, really, really" and stop. It followed the rule, but the answer was nonsense.
The Fix: To stop this, the researchers added a "Coherence Score." They used another AI to grade if the story actually made sense. If the robot wrote nonsense just to follow the rules, its score went down.

The Bottom Line

CAPITU is a new, fair, and culturally aware way to test if AI can actually listen to us when we speak Portuguese.

Good News: The smartest AI models are getting very good at following strict rules.
Bad News: Many models still struggle with the unique "flavors" of Portuguese (like word endings) and tend to forget rules during long conversations.
Big Takeaway: You don't always need the biggest, most expensive AI to get the job done in Portuguese; sometimes, a specialized, smaller model is the better, cheaper choice.

The researchers have released all their test questions and code for free, so other scientists can help build better, more obedient AI assistants for the Portuguese-speaking world.

1. Problem Statement

While Large Language Models (LLMs) have shown strong instruction-following capabilities in English, evaluation in Brazilian Portuguese (PT-BR) remains underexplored. Existing multilingual benchmarks (e.g., Multi-IF) rely heavily on translating English prompts, which fails to capture:

Linguistic Specificity: Portuguese has unique morphological features (e.g., productive diminutive suffixes -inho/-zinha, gerund forms -ando/-endo/-indo, and adverbial -mente) that do not have direct English equivalents.
Cultural Context: Translated prompts often miss culturally grounded discourse markers, rhetorical norms, and literary nuances specific to Brazil.
Evaluation Authenticity: Translation artifacts can introduce constraints that are unnatural or irrelevant to the target language.

There is a critical need for a native, automatically verifiable benchmark that tests instruction-following within the rich cultural and linguistic context of Brazilian literature.

2. Methodology

CAPITU (Contextual Assessment of Portuguese Instruction-following Through Literary Understanding) is designed with four core principles: Automatic Verification, Linguistic Relevance, Cultural Grounding, and Scalable Difficulty.

A. Instruction Taxonomy

The benchmark defines 59 instruction types across 7 categories, all designed for deterministic verification (using string operations, regex, and lexicon matching) without human judges:

Count: Word, sentence, paragraph, and character counts.
Words: Keyword inclusion, frequency, and specific conjunctions.
Pattern (PT-Specific): Constraints on word endings (e.g., -ando/-endo/-indo, -inho/-inha, -ão/-ões, -mente).
Forbidden: Prohibiting specific words, numbers, first-person pronouns, or questions.
Structure: Acrostics, line prefixes, start/end word constraints.
Punctuation: Specific punctuation usage (e.g., semicolons, quotes).
Format: Lists, capitalization rules.

B. Literary Corpus & Context

Unlike generic benchmarks, CAPITU contextualizes all prompts within 8 canonical works of Brazilian literature (e.g., Dom Casmurro, Grande Sertão: Veredas, Macunaíma), spanning movements from Romanticism to Modernism. This ensures prompts are coherent and culturally relevant.

C. Evaluation Framework

Dataset Size: 500 total instances.
- 200 Single-turn prompts: Varied by difficulty (Easy: 1 constraint; Medium: 2; Hard: 3–4).
- 100 Multi-turn conversations: 3 turns each (300 total turns), testing constraint persistence as instructions accumulate.
Metrics:
- Strict Accuracy: Exact match of all constraints.
- Loose Accuracy: Allows for minor formatting artifacts (e.g., markdown headers).
- Coherence Score: An optional LLM-based judge (GPT-4o-mini) evaluates content quality to penalize "degenerate" solutions (e.g., repeating a word 80 times to meet a count constraint).
- Final Score: A weighted combination ( $2 \times \text{Instruction Compliance} + \text{Coherence}$ ) / 3.

3. Key Contributions

First Native PT-BR Instruction Benchmark: Introduces 59 verifiable instruction types specifically designed for Portuguese morphology and syntax, moving beyond translation-based approaches.
Cultural Grounding: Integrates prompts with 8 major Brazilian literary works, providing a realistic and diverse linguistic environment.
Automated & Reproducible Evaluation: Eliminates the need for LLM judges or human annotators for constraint checking, ensuring objective, scalable, and deterministic results.
Comprehensive Baseline: Evaluates 18 state-of-the-art models (including OpenAI, Google, Anthropic, Alibaba, and Portuguese-specialized models) across single-turn and multi-turn settings.
Open Release: The full benchmark, evaluation code, and generation tools are open-sourced.

4. Key Results

The paper evaluated 18 models, including frontier reasoning models and Portuguese-specialized models.

Top Performers:
- GPT-5.2 (Reasoning): Achieved the highest strict accuracy (98.5%), demonstrating that reasoning capabilities significantly boost instruction adherence.
- GPT-5 Family: Consistently outperformed other families, with minimal degradation between easy and hard constraints.
Portuguese-Specialized Models:
- Sabiazinho-4 (Maritaca AI) achieved 87.0% strict accuracy, outperforming Claude-Sonnet-4.5 (86.0%) and Claude-Haiku-4.5 (73.5%).
- Cost-Efficiency: Sabiazinho-4 was significantly more cost-effective ($0.13 vs. $1.12 for Claude-Haiku) while delivering superior performance.
Difficulty & Degradation:
- Performance drops significantly as constraints increase. Models like Claude-Haiku-4.5 and Qwen3-8B showed massive drops (30–40%) from Easy to Hard questions.
- Multi-turn Persistence: Conversation-level accuracy ranged widely (60%–96%). While GPT-5-mini maintained 96% across turns, many models struggled to retain constraints by Turn 3.
Category Analysis:
- Easiest: "Forbidden" constraints (99.1% avg).
- Hardest: Exact word counts (21.1% avg) and structural constraints like acrostics.
- Morphological Challenges: Portuguese-specific patterns (e.g., -inho/-inha) remained difficult for non-GPT models.

5. Significance

CAPITU addresses a critical gap in NLP research by providing a culturally and linguistically grounded evaluation suite for Portuguese. Its significance lies in:

Validating Native Capabilities: Proving that translation-based benchmarks are insufficient for capturing the nuances of Portuguese instruction following.
Highlighting Reasoning Benefits: Demonstrating that "thinking" or reasoning modes are crucial for handling complex, multi-constraint tasks in Portuguese.
Cost-Performance Insights: Showing that smaller, domain-specialized models (like Sabiazinho-4) can outperform larger, generalist models in specific linguistic contexts at a fraction of the cost.
Future Research: Providing a foundation for improving multi-turn constraint persistence and morphological control in LLMs, with open tools to accelerate Portuguese NLP development.

The authors conclude that while frontier models are approaching near-perfect instruction following in Portuguese, significant challenges remain in morphological control, exact counting, and long-context constraint persistence.