Technical Summary: JurisTCU – A Brazilian Portuguese Information Retrieval Dataset with Query Relevance Judgments

Problem Statement

The field of Legal Information Retrieval (LIR) faces a significant bottleneck in the Portuguese-speaking domain: a scarcity of high-quality datasets containing both a legal corpus and query relevance judgments (qrels). While general Portuguese IR datasets exist (e.g., Quati) and some legal datasets are available for English (e.g., CLERC, MS MARCO), there is a notable absence of Portuguese LIR datasets that include typical search system queries annotated with relevance labels. Existing Portuguese legal resources, such as the Ulysses-RFCorpus, often feature query formats (e.g., legislative requests for new bills) that diverge from standard user search behaviors. This lack of standardized evaluation resources hinders the development and benchmarking of search systems for Brazilian citizens and legal professionals, particularly for the Tribunal de Contas da União (TCU), which manages over 500,000 jurisprudential documents.

Methodology

The authors developed JurisTCU, a dataset comprising 16,045 jurisprudential documents and 150 queries with relevance judgments. The methodology involved three primary stages:

1. Corpus Construction

The document corpus was derived from the TCU's "selected jurisprudence" collection, a curated subset of Acórdãos (official judicial decisions) chosen for their precedential value. Each document in the dataset includes metadata and two primary text fields:

• SUMMARY (ENUNCIADO): A concise statement of the legal understanding (avg. 306 characters).
• EXCERPT (EXCERTO): A textual segment from the original decision providing the basis for the summary (avg. 4,387 characters).

2. Query Generation

The 150 queries were organized into three distinct groups to simulate different search behaviors:

• Group G1 (Real User Queries): The 50 most frequent queries from the TCU's search logs (June 2022–May 2023). These are short, keyword-based queries (avg. 3.5 words) without qualifiers or prepositions.
• Group G2 (Synthetic Keyword Queries): 50 queries generated by prompting an LLM (ChatGPT-4) to extract essential keywords from the summaries of the 50 most accessed documents. These average 6.5 words.
• Group G3 (Synthetic Question Queries): 50 full-text questions generated from the same summaries, refined manually to be representative. These average 16.5 words.

3. Relevance Judgment (Qrels) Annotation

To address the high cost of manual annotation, the authors employed a hybrid two-step process:

1. Automated Pre-selection: A two-stage retriever (combining BM25 and a BERT-based semantic retriever) selected the top 300 candidate documents per query. These were reranked using a T5-based model, and the top 10 results were selected. Additionally, 5 random documents from the BM25 output were included to ensure non-relevant examples.
2. LLM Scoring & Expert Validation: The 15 selected documents per query were scored by GPT-4 on a scale of 0 (irrelevant) to 3 (highly relevant). A domain expert then reviewed these scores to correct any errors. In the JurisTCU case, the expert largely agreed with the automated scores, resulting in 2,250 total relevance judgments.

Experimental Setup

The authors conducted 14 experiments to establish benchmarks for the dataset, comparing a standard BM25 baseline against lexical and semantic retrieval approaches:

• Lexical Search: Standard BM25 versus BM25 with Document Expansion. Expansion methods included:
  • docT5query: Generating synthetic queries from the summary using a Portuguese T5 model.
  • Synonym Expansion: Using LLMs (GPT-3.5, GPT-4o, Llama 3-70B) to generate synonyms for key terms in the summary.
  • Combinations: Merging both expansion techniques.
• Semantic Search: Dense vector retrieval using embeddings from:
  • BERT-based models: Including multilingual models and Portuguese-specific models (general and legal domain fine-tuned).
  • OpenAI models: text-embedding-3-small and text-embedding-3-large.

Performance was evaluated using Precision@10 (P@10), Recall@10 (R@10), Mean Reciprocal Rank (MRR@10), and nDCG@10.

Key Results

Lexical Search Performance

Document expansion techniques significantly improved retrieval performance over the baseline, particularly for short, keyword-based queries (Group G1).

• Improvement Magnitude: For Group G1, combining docT5query with synonym expansion (using GPT-3.5 or Llama 3) increased P@10, R@10, and nDCG@10 by over 45% compared to the baseline.
• Model Comparison: The docT5query method generally outperformed synonym-only expansion. Among LLMs, GPT-3.5 and Llama 3-70B yielded similar, superior results compared to GPT-4o for this specific task.
• Query Length Impact: As queries became longer and more aligned with document vocabulary (Groups G2 and G3), the baseline BM25 performance increased, making improvements from expansion more modest (approx. 8% for P@10/R@10 in Group G3).

Semantic Search Performance

The results highlighted a stark divergence between model families:

• BERT-based Models: All BERT-based embedding models (both multilingual and Portuguese legal fine-tuned) underperformed the standard BM25 baseline across all query groups. The authors attribute this to potential information loss due to token truncation and the difficulty of semantic models in handling short, keyword-heavy queries without specific fine-tuning on this dataset.
• OpenAI Models: OpenAI embedding models consistently achieved the best results, surpassing both the BM25 baseline and the best lexical expansion methods.
  • For Group G1 (short queries), OpenAI models improved P@10, R@10, and nDCG@10 by approximately 70%.
  • The larger model (text-embedding-3-large) generally outperformed the smaller one (text-embedding-3-small) in Groups G1 and G2, though the smaller model performed slightly better in Group G3.
  • These results suggest that dense embeddings effectively capture semantic relationships in the legal domain, even for short queries, surpassing reliance on exact lexical matching.

Significance and Contributions

The paper claims three primary contributions:

1. Dataset Availability: JurisTCU fills a critical gap in the Portuguese IR research community by providing the first LIR dataset with typical search queries and relevance judgments. It enables the benchmarking of retrieval systems in a specialized legal context.
2. Empirical Insights: The study provides evidence that while document expansion significantly boosts lexical search (especially for short queries), semantic search using high-quality embeddings (specifically OpenAI models) offers superior performance, even outperforming expanded lexical methods. Conversely, it warns that standard BERT-based models may not be suitable for this domain without further fine-tuning.
3. Practical Impact: The dataset and experimental results offer a pathway to improve the TCU's public search system, which is accessed over 1.5 million times annually, thereby enhancing access to legal precedents for Brazilian citizens and professionals.

The authors conclude that while the dataset is specific to TCU jurisprudence, it serves as a valuable resource for studying the impact of new embedding models and search techniques in the Portuguese language. Future work involves releasing diverse annotated datasets and exploring manual annotation to compare with automatic judgments.