The Effect of Document Selection on Query-focused Text Analysis

1. Problem Statement

Large-scale text corpus analysis (e.g., topic modeling, qualitative coding) often requires selecting a subset of documents due to computational constraints and the fact that most documents in a large collection are irrelevant to a specific research question.

• The Gap: While data selection is a practical necessity, it is currently treated as a "black box" or a simple heuristic (e.g., random sampling or basic keyword filtering) rather than a deliberate methodological decision.
• The Risk: Poor selection strategies can lead to:
  • Irrelevant Findings: Including too much noise (irrelevant documents).
  • Limited Discovery: Missing nuanced themes due to over-optimizing for relevance (reducing diversity).
  • Wasted Resources: Using expensive retrieval methods that offer no marginal benefit over simpler ones.
• The Question: How do different document selection strategies (from random sampling to advanced semantic retrieval) impact the quality, relevance, diversity, and coverage of the final topics discovered by various text analysis methods?

2. Methodology

The authors propose a systematic evaluation framework to assess the impact of data selection on downstream analysis.

Datasets & Queries

• TREC-COVID: 171,332 biomedical articles with 15 expert-designed open-ended queries (e.g., "impact of COVID on violence").
• Doctor-Reviews: 171,216 physician reviews from Healthgrades, Vitals, etc., with 11 expert-designed queries (e.g., "patient experiences with referrals").
• Constraint: All strategies select exactly 1,000 documents per query to isolate the effect of which documents are chosen, not how many.

Data Selection Strategies (7 Evaluated)

The study compares a spectrum of retrieval approaches:

1. Random Uniform: Baseline random sampling.
2. Keyword Search (BM25): Traditional lexical matching.
3. SBERT: Dense semantic retrieval using embeddings.
4. Direct Retrieval (Hybrid Simple Sum): Combines BM25 and SBERT scores equally.
5. Hybrid RRF: Reciprocal Rank Fusion of BM25 and SBERT.
6. Query Expansion (KeyBERT): Expands queries with extracted keywords before retrieval.
7. MMR (Maximal Marginal Relevance): Enforces diversity by trading off relevance to reduce redundancy.
   (Note: Cross-Encoder reranking and Retrieval Random were tested but excluded from final analysis due to cost or lack of benefit.)

Downstream Analysis Methods (4 Models)

The selected documents are processed by four distinct topic modeling approaches:

1. LDA: Statistical, bag-of-words approach.
2. BERTopic: Clustering-based approach using semantic embeddings.
3. TopicGPT: LLM-based prompt engineering for topic generation.
4. HiCode: Hierarchical inductive coding using LLMs (query-aware).

Evaluation Metrics

Since ground-truth topics do not exist for open-ended queries, the authors define three metrics:

1. Relevance: Cosine similarity between the query embedding and the generated topic embeddings.
2. Diversity: Average pairwise distance between topic embeddings (measuring distinctness).
3. Coverage: Pairwise overlap analysis. Does Strategy A cover the relevant topics found by Strategy B? (Measured via Hungarian algorithm matching at a similarity threshold of 0.7).

3. Key Results

Relevance

• Semantic/Hybrid > Lexical > Random: Semantic methods (SBERT, Direct Retrieval, Query Expansion) consistently outperform Keyword Search and Random Uniform in generating topics relevant to the query.
• Model Sensitivity:
  • HiCode shows uniformly high relevance across all selection methods because its architecture explicitly incorporates the query text into the LLM prompts, effectively "forcing" relevance regardless of the input data.
  • TopicGPT and BERTopic show the most significant performance gaps between selection strategies.

Diversity vs. Relevance Trade-off

• Inverse Relationship: There is a strong negative correlation between relevance and overall diversity. High-relevance strategies (semantic retrieval) yield lower overall diversity than random sampling.
• The "Noise" Factor: The high diversity of Random Uniform is driven primarily by irrelevant/off-topic content. When diversity is calculated only among query-relevant topics (similarity > 0.5), the differences between selection strategies vanish.
• Implication: Semantic retrieval filters out noise while preserving diversity where it matters (among relevant topics).

Coverage

• Convergence of Semantic Methods: SBERT, Direct Retrieval, and Query Expansion converge on similar sets of relevant topics. They cover each other's findings significantly (50–68% overlap).
• Random Uniform Failure: Random sampling covers a negligible fraction (4–9%) of the relevant topics found by semantic methods.
• Keyword Search Artifacts: Keyword search suffers from polysemy. In a case study ("violence in society"), it retrieved ~30% of documents about "surgical societies" due to the word "society," introducing substantial noise that semantic methods avoided.

Advanced Methods (Query Expansion & MMR)

• Query Expansion: Offers minimal improvement in relevance over Direct Retrieval but adds computational overhead.
• MMR: While it increases diversity, it often reduces relevance significantly and introduces high variance. It is only recommended when diversity is the primary goal and resources allow for tuning.

4. Key Contributions

1. Methodological Framework: Establishes data selection as a critical, evaluable methodological choice rather than a mere preprocessing step.
2. Systematic Evaluation: Provides the first large-scale comparison of selection strategies across diverse retrieval methods and four distinct topic modeling architectures.
3. Practical Guidelines:
   • Default Recommendation: Use SBERT (for ease of implementation) or Direct Retrieval (Hybrid) for the best balance of relevance and diversity.
   • Avoid: Random sampling for query-focused analysis (leads to noise) and Keyword Search for open-ended queries (prone to polysemy errors).
   • Avoid: Query Expansion and Cross-Encoders for this specific task, as they offer marginal gains at high cost.
4. Insight on Diversity: Demonstrates that "sacrificing relevance for diversity" is often a false trade-off; semantic retrieval maintains diversity among relevant topics while removing irrelevant noise.

5. Significance

This work fundamentally shifts the paradigm of corpus analysis. It argues that researchers cannot simply "sample randomly" and assume the results are representative. The choice of selection strategy fundamentally dictates which themes can be discovered.

• For Researchers: It provides an evidence-based protocol for selecting data, ensuring that computational resources are spent on analyzing relevant, diverse, and high-quality subsets of data.
• For the Field: It highlights that classical Information Retrieval (IR) metrics (Precision/Recall) are insufficient for evaluating data selection for exploratory analysis, necessitating new metrics that account for thematic diversity and coverage.
• Future Work: The framework invites the development of new selection strategies specifically optimized for the goals of exploratory text analysis rather than just document retrieval.