Technical Summary: HieraRAG – A Hierarchical Framework for Synthetic Question Generation

Problem Statement

Evaluating Retrieval-Augmented Generation (RAG) systems requires benchmarks that capture diverse question characteristics. However, practitioners lack empirical guidance on which dimensions of question variation to include and, crucially, at what level of granularity (e.g., coarse vs. fine categories) these dimensions should be defined. Existing benchmarks often apply inconsistent granularity: some categorize by answer type but lack complexity control, while others span task formats without systematic linguistic variation. Without empirical evidence, designers risk either under-sampling critical failure modes or diluting evaluation signals with redundant, fine-grained distinctions that increase generation costs without adding diagnostic value.

Methodology

The authors propose HieraRAG, a hierarchical framework designed to systematically study the impact of question granularity on RAG evaluation. The methodology involves three core components:

1. Hierarchical Categorization Framework

The framework defines three illustrative dimensions of question variation, each evaluated at three granularity levels:

• Question Complexity (QC): Cognitive demand ranging from fact lookup to multi-hop reasoning.
• Answer Type (AT): Expected response format, from direct extraction to synthesized generation.
• Linguistic Variation (LV): Lexical alignment between question and document, from verbatim to paraphrased.

Each dimension is subdivided into Coarse (2 categories), Medium (4 categories), and Fine (8 categories) levels. For example, Question Complexity moves from "Simple/Complex" (Coarse) to "Single-fact/Multi-fact/Cross-section/Comparative" (Medium) to specific types like "Extractive span," "Multi-hop reasoning," and "Comparative synthesis" (Fine).

2. Synthetic Data Generation and Experimental Setup

• Data Source: 5,872 synthetic question-answer (QA) pairs were generated from the FineWeb-10BT corpus using DataMorgana (leveraging Claude 3.5 Sonnet).
• RAG Pipeline: The system was evaluated on a standard two-stage pipeline:
  • Retrieval: BM25 (via PyTerrier) retrieving $k=10$ documents from 512-token chunks.
  • Generation: Falcon-3-10B-Instruct (temperature 0.6) instructed to refuse if information was insufficient.
• Experimental Design:
  • RQ1: Evaluated 4 dimensions (QC, AT, LV, and Phrasing) at coarse granularity to identify the most discriminative dimensions.
  • RQ2: Evaluated QC, AT, and LV across all three granularity levels (2, 4, 8 categories) to determine optimal granularity.
  • RQ3: A $2 \times 2$ factorial design crossing Linguistic Variation (Similar vs. Distant) with Question Complexity (Simple vs. Complex) to test interactions.

3. Evaluation Metrics

• Performance Metrics: Mean Average Precision (MAP), nDCG@10, Recall@10 for retrieval; Cosine Similarity (CS), ROUGE-1, and BLEU for generation quality.
• Discriminative Power: Defined as the standard deviation of generation quality (CS) across categories within a granularity level. Higher standard deviation indicates the categories reveal meaningful performance distinctions.
• Coherence Ratio ($\rho_{coherence}$): A novel metric inspired by clustering silhouette coefficients to validate hierarchical structure. It measures whether fine-grained "sibling" categories provide non-redundant signals while remaining vertically consistent with their parent category:
  $$\rho_{coherence} = \frac{\sigma_{sib}}{\delta_{vert} + \epsilon}$$
  Where $\sigma_{sib}$ is the standard deviation across siblings (horizontal discrimination) and $\delta_{vert}$ is the absolute difference between parent and child means (vertical consistency). A high $\rho$ indicates well-calibrated splits; a low $\rho$ suggests poor structure or redundancy.
• Human Validation: Two annotators evaluated 110 stratified QA pairs for correctness, hallucination, and category alignment.

Key Results

RQ1: Most Discriminative Dimensions

At the coarse level, Linguistic Variation (LV) was the most discriminative dimension (Range = 0.077), despite yielding the worst absolute performance (MAP=0.369). Answer Type (AT) achieved the best absolute performance (MAP=0.506) but showed lower discrimination. Question Complexity (QC) showed the lowest discrimination at the coarse level (Range = 0.010).

RQ2: Impact of Granularity

Optimal granularity varies significantly by dimension:

• Question Complexity (QC): Benefits from fine-grained distinctions. Discriminative power increased monotonically from Coarse (0.007) to Fine (0.053), a 7.5x increase. This suggests that distinguishing between specific reasoning types (e.g., single-hop vs. multi-hop) captures performance variations missed by coarse "Simple/Complex" labels.
• Answer Type (AT) & Linguistic Variation (LV): Peak at medium granularity (4 categories).
  • AT peaked at 0.044 (Medium) and declined to 0.037 (Fine).
  • LV peaked at 0.047 (Medium) and declined to 0.031 (Fine).
  • For these dimensions, finer splits added noise rather than signal.

Structural Validation (Coherence Ratio)

The Coherence Ratio revealed structural differences explaining the results:

• AT had a high mean coherence (e.g., the summary_or_explanation split had $\rho=3.31$), indicating that its fine categories cleanly subdivided the parent.
• QC had a low mean coherence (0.40). While the 8 fine categories successfully partitioned the performance space (high discriminative power), they did not cleanly subdivide their medium-level parents, suggesting the hierarchy is not strictly taxonomic.
• LV showed poor calibration in the conceptual_rephrase split ($\rho=0.04$), where fine children strayed substantially from the parent's performance profile.
• A preliminary correlation ($r_{pb}=0.24$) was observed between the Coherence Ratio and human agreement on category assignments, suggesting the metric may serve as an automated proxy for semantic separation.

RQ3: Interaction Effects

The interaction between LV and QC was largely additive rather than interactive.

• Vocabulary mismatch (Distant LV) caused a significant drop in performance regardless of complexity (MAP dropped from ~0.55 to ~0.20).
• Complexity distinctions only mattered when retrieval was successful (Similar LV). When retrieval failed due to vocabulary mismatch, complexity became irrelevant.

Significance and Claims

The paper claims to provide a portable procedure rather than universal truths about RAG performance. Its primary contributions are:

1. A Hierarchical Framework: A method for determining optimal evaluation granularity in RAG benchmarks, demonstrating that a "one-size-fits-all" granularity is suboptimal.
2. The Coherence Ratio: A validation metric to help practitioners determine if fine-grained splits are structurally sound before large-scale benchmark generation.
3. Empirical Evidence: Within the specific configuration of BM25 + Falcon-3-10B, the authors show that different question characteristics require different granularity levels for diagnostic evaluation (Fine for Complexity, Medium for Answer Type and Linguistic Variation).
4. Dataset and Code: A release of 5,872 hierarchically organized synthetic questions and the associated code for replication.

The authors explicitly note limitations: the findings are specific to the BM25 retriever and Falcon generator used; the single-dimension assignment in RQ1/RQ2 leaves other dimensions as uncontrolled confounders; and the correlation between the Coherence Ratio and human agreement was not statistically significant ($p=0.21$). Consequently, the paper positions HieraRAG as a tool for practitioners to diagnose and calibrate their own RAG evaluation settings rather than prescribing a fixed benchmark standard.