Technical Summary: SPECTRA – Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics

Problem Statement

Scalable information retrieval (IR) testing requires corpora large enough to stress index construction, ranking latency, query routing, and evaluation tooling. However, existing human-judged test collections (e.g., TREC, BEIR, MS MARCO) are expensive to produce, often unavailable for private or pre-production domains, and cannot be easily modified to test specific engineering constraints. While prior synthetic corpus work has focused on statistical properties (e.g., document length, Zipfian distributions) or using Large Language Models (LLMs) to generate text, there is a gap in generating large-scale synthetic test collections that retain auditable truth conditions, graded relevance labels, and controlled failure diagnostics without relying on opaque LLM consistency.

Methodology: The SPECTRA Framework

The paper introduces SPECTRA (Synthetic Probabilistic Evaluation Corpus for Text Retrieval Assessment), a reproducible framework that separates the generation of latent structure from surface text realization. The architecture consists of five distinct layers:

1. Latent Layer: Samples topics, entities, dates, and document roles. It defines the "ground truth" before any text is generated.
2. Surface Layer: Converts latent assignments into text. In the prototype, this uses synthetic lexical tokens rather than natural sentences to prioritize indexing and ranking behavior over human readability.
3. Query Layer: Samples information needs and query formulations based on topic anchor terms and optional metadata intents.
4. Oracle Layer: A deterministic module that maps query-document pairs to graded relevance labels (0–3) based on latent topic matches and anchor term overlap, independent of the surface text realization.
5. Export Layer: Outputs documents, queries, qrels, and diagnostics in standard formats (JSONL, TSV) compatible with retrieval stacks like Lucene and Pyserini.

Key Design Choices:

• Deterministic Oracles: Relevance is defined by latent facts, ensuring labels are auditable and reproducible, avoiding the "black box" nature of LLM-generated relevance.
• Controlled Distractors: A tunable parameter injects cross-topic vocabulary into documents to create lexical ambiguity and stress-test ranking assumptions.
• Metadata Control: Documents include high-cardinality categorical fields (genre, date, visibility) to test faceting, filtering, and routing.

Experimental Setup and Results

A single-process Python prototype was used to conduct a local simulation study. The setup involved 32 latent topics, a 120,000-token vocabulary, and 96 queries. Two primary experiments were conducted:

1. Scaling Experiment

The system generated corpora of 5,000, 20,000, and 60,000 documents (up to 9.61 million tokens) with a fixed 12% distractor rate.

• Throughput: Generation remained near-linear, processing approximately 12,000 to 14,000 documents per second.
• Corpus Statistics: The system preserved controllable long-tail vocabulary growth. The estimated Zipf slope remained stable at approximately 0.86 in absolute value across scales.
• Retrieval Performance: BM25 nDCG@10 remained high (0.986–0.996) across scales, indicating the baseline retrieval logic functioned correctly on the synthetic data.

2. Difficulty Experiment

Using a 20,000-document corpus, the distractor rate was varied from 2% to 36% to test retrieval robustness.

• BM25 Degradation: As distractor text increased, BM25 nDCG@10 dropped from 0.999 (at 2% distractors) to 0.434 (at 36% distractors).
• Baseline Comparison: At 24% distractors, a raw term-frequency (TF) baseline outperformed BM25. The authors note this is not a contradiction but a diagnostic insight: in high-distractor settings, repeated anchor terms become artificially strong signals, while BM25's length normalization and term-frequency saturation reduce the reward for repetition.
• Recall Behavior: Recall@100 decreased as the corpus scaled, demonstrating that metric changes can result from collection size shifts (more relevant documents per query) rather than ranking system failures.

Key Contributions

The paper contributes:

1. A Requirements Model: Defining the necessary components for synthetic IR test collections, emphasizing auditability and known relevance.
2. The SPECTRA Architecture: A separation of latent facts, metadata, surface text, and graded relevance, allowing for tunable stress factors.
3. Reproducible Prototype & Diagnostics: A local simulation demonstrating linear scaling and the ability to expose retrieval failure modes (e.g., scoring assumptions) through controlled distractors.
4. Practical Guidance: A framework for using synthetic corpora as a "unit and stress test" for retrieval infrastructure before moving to expensive human-judged benchmarks.

Significance and Claims

The authors position SPECTRA not as a replacement for human assessment but as a diagnostic complement to Cranfield-style and TREC-style evaluation. The paper makes three modest claims supported by the results:

1. Scalability: Deterministic simulators can generate millions of tokens quickly enough for repeated parameter sweeps and index sizing.
2. Transparency: Latent-oracle designs provide cheap relevance labels without obscuring the reasons for those labels, enabling repeatable ablation studies.
3. Stress Testing: Controlled distractors offer a direct method to stress-test ranking assumptions (e.g., how BM25 handles lexical ambiguity) before costly collection construction begins.

The paper concludes that synthetic corpora are most valuable when used for staged evaluation: verifying infrastructure (indexing, sharding, filtering, metric computation) at target scale with SPECTRA, followed by validation on public or private human-judged collections. The framework acknowledges limitations in surface realism (using tokens rather than natural language) and suggests future work should integrate template or LLM realization modes while maintaining the separation between the latent oracle and surface text to ensure verifiability.