Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese

Technical Summary: Prosa – Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese

Problem Statement
Current automatic evaluation of open-ended language generation faces two primary limitations. First, traditional n-gram overlap metrics (e.g., BLEU, ROUGE) correlate weakly with human judgment, particularly failing to distinguish between high-quality systems. Second, while "LLM-as-a-judge" paradigms have improved correlation with human experts, they rely on holistic scoring, where a judge collapses multiple quality dimensions into a single score or pairwise preference. The authors argue that this holistic approach is opaque and highly sensitive to the specific biases of the chosen judge model. Furthermore, existing benchmarks for Brazilian Portuguese are dominated by academic multiple-choice exams or translated English tasks, lacking evaluation based on naturally occurring, multi-turn user interactions within the specific cultural and linguistic context of Brazil.

Methodology
The paper introduces Prosa, a benchmark and evaluation protocol designed to address these gaps through three core design choices:

1. Real-World Data Construction: Prosa derives its dataset from the WildChat repository, filtering 3.2 million conversations down to 1,000 high-quality, multi-turn interactions in Brazilian Portuguese. The filtering pipeline (Figure 1) employs rule-based checks (IP uniqueness, token length, semantic deduplication) and LLM-based filters (difficulty classification, topic assignment, and nonsensical intent detection) to ensure the data reflects verifiable, non-redundant, and challenging user tasks.
2. Decomposed Rubric-Based Scoring: Instead of holistic scoring, Prosa utilizes a binary pass/fail rubric system. For each conversation, a generator model (GPT-5.2) produces a set of atomic binary criteria based on the conversation history and reference responses. A judge model then evaluates each response against these criteria, emitting a YES/NO verdict for each rubric. The final score is the Decomposed Requirements Following Ratio (DRFR), scaled to 0–100.
3. Multi-Judge Post-Hoc Filtering: A novel pipeline filters the generated rubrics before scoring to remove low-quality criteria. This pipeline operates on the empirical ranking of 16 candidate models evaluated by three distinct judges (GPT-4.1, Gemini 3 Flash, and Sabiá-4). It removes rubrics that are:
   • Trivial: Passed by all candidates.
   • Impossible: Failed by all candidates.
   • Misaligned: Where top-scoring candidates fail while bottom-scoring candidates pass.
   • Unstable: Where verdicts change across repeated runs at temperature 0.

Key Contributions

• Prosa Benchmark: The first real-user, multi-turn Brazilian Portuguese chat benchmark evaluated via rubric-based LLM-as-a-judge.
• Methodological Insight: The demonstration that decomposing judgment into atomic binary verdicts is more critical for ranking stability than the choice of the judge model itself.
• Filtering Pipeline: An automatic, post-hoc rubric filtering mechanism that significantly improves the discriminative power of the evaluation without requiring a "frontier" judge model.
• Cost Efficiency: A protocol that allows high-fidelity evaluation using cheaper models (e.g., Gemini 3 Flash at ~$2.1 per candidate) while reproducing rankings obtained by more expensive judges.

Results
The authors evaluated 16 models from four families (OpenAI, Google, Alibaba, Maritaca AI) using three different judges under three protocols: holistic scoring, unfiltered rubric scoring, and filtered rubric scoring.

• Rank Stability: Under holistic scoring, the three judges agreed on only 7 of 16 ranks (Spearman correlation 0.986). Under filtered rubric scoring, the judges achieved perfect agreement (16/16 ranks, Spearman 1.000).
• Discriminative Power: The filtering pipeline increased the average score gap between neighboring models by 47% (from 2.22 to 3.26 points) and expanded the top-to-bottom spread from 33.3 to 48.9 points.
• Model Performance: The final leaderboard (Table 4) shows GPT-5.2 at the top (88.6), followed closely by GPT-5 Mini and Gemini 3 Pro. Notably, the open-weights Qwen3-235B (80.1) outperformed GPT-4.1 and Gemini 3 Flash, ranking 4th. The Maritaca AI model Sabiá-4 ranked 9th (70.2), showing a significant generational improvement over Sabiá-3.1.
• Cost: Evaluating a new candidate costs approximately $2.1 using Gemini 3 Flash, an order of magnitude cheaper than comparable benchmarks like Arena-Hard-Auto ($20/candidate).

Significance and Claims
The paper claims that the primary driver of reliable LLM evaluation is the decomposition of the judgment into atomic binary decisions rather than the sophistication of the judge model. By shifting from holistic scoring to filtered rubric scoring, the evaluation becomes invariant to the specific judge used, allowing researchers to use cheaper, faster models without sacrificing ranking fidelity.

The authors posit that this methodology lowers the barrier for large-scale evaluation and provides fine-grained quality signals (per-rubric verdicts) that could be repurposed for reward modeling in addition to evaluation. They conclude that while the benchmark is specific to Brazilian Portuguese, the core mechanism of decomposed, filtered rubric evaluation is likely transferable to other languages and open-ended evaluation settings. The authors release the benchmark, filtering code, and rubrics to ensure reproducibility and identical evaluation conditions for future models.