RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competition

Based on the provided paper, here is a detailed technical summary of the RMIT-ADM+S system and its participation in the NeurIPS 2025 MMU-RAG Competition.

1. Problem Statement

The paper addresses the challenge of building Retrieval-Augmented Generation (RAG) systems that are not only accurate but also adaptive, resource-efficient, and user-centric. Traditional RAG systems often apply a uniform retrieval strategy to all queries, which can be inefficient for simple questions or insufficient for complex, multi-step research tasks. The specific context was the MMU-RAG NeurIPS 2025 Competition, which emphasized:

• Dynamic Evaluation: Systems were tested in real-time "RAG-Arena" settings where user preferences and qualitative feedback mattered more than static aggregate metrics.
• Resource Constraints: The goal was to achieve high performance on consumer-grade hardware (single GPU) rather than relying on massive, expensive clusters.
• Complexity Handling: The need to distinguish between simple factoid queries and complex queries requiring deep reasoning, multi-perspective synthesis, and iterative search.

2. Methodology: Routing-to-RAG (R2RAG)

The core contribution is R2RAG, a dynamic architecture that routes queries to different processing pipelines based on inferred complexity. The system is built upon the G-RAG system (winner of SIGIR 2025 LiveRAG) but introduces significant refinements based on qualitative user studies.

A. System Architecture

The workflow consists of three main stages:

1. Query Classification:

   • An LLM-based classifier (using Qwen3-4B) determines if a query is "Simple" or "Complex."
   • Definition: Complex queries are non-factoid questions requiring long-form reasoning, multi-step synthesis, or multiple search steps.
   • Decision: The classifier routes the query to either a Vanilla RAG pipeline (simple) or a Vanilla Agent pipeline (complex).

2. Pipeline 1: Vanilla RAG (for Simple Queries)

   • Single-pass retrieval: Generates 3 query variants to broaden coverage.
   • Retrieval: Searches the ClueWeb22-A index in parallel.
   • Reranking: Uses a lightweight Qwen3-Reranker-0.6B model to score documents.
   • Generation: Synthesizes an answer from the top documents (truncated to 5k words) using Qwen3-4B.

3. Pipeline 2: Vanilla Agent (for Complex Queries)

   • Iterative Search Loop: Operates in a loop (max 5 iterations) to gather complementary evidence.
   • State Management: Accumulates used queries (to avoid repetition) and summaries of useful documents.
   • Dynamic Query Reformulation: An LLM-based Document Review module analyzes the current evidence against the query. It checks for:
     • Coverage of multiple perspectives.
     • Completion of sub-parts.
     • Balance on controversial topics.
     • Identification of knowledge gaps.
   • Termination Conditions: The loop stops if:
     • Token count > 20,000 (protecting context window).
     • Information coverage is deemed sufficient (by the LLM).
     • Maximum iterations (5) are reached.
   • Generation: Once sufficient evidence is gathered, the LLM generates a comprehensive response with inline citations.

B. Model Selection & Resource Efficiency

• Models: The system relies entirely on Qwen3 models:
  • Qwen3-4B for classification, query generation, document review, and answer generation.
  • Qwen3-Reranker-0.6B for document reranking.
• Hardware: The entire system runs on a single consumer-grade GPU.
• Optimization:
  • Models are used without quantization (unlike larger 8B+ models which degraded in performance when quantized in preliminary tests).
  • Context window reduced to 25,000 tokens (from native 32k) to balance memory usage and performance.
  • vLLM is used for efficient inference serving.

C. Qualitative Evaluation & Refinement

A critical part of the methodology was a qualitative focus-group study involving 20 participants (students to professors) from RMIT University.

• Findings: Quantitative metrics failed to capture user preferences regarding verbosity, tone, and structure.
  • Vanilla Agent was better for complex queries but too verbose for simple ones.
  • Vanilla RAG was concise for simple queries but failed on multi-faceted tasks.
• Impact: These insights directly informed the routing strategy and prompt engineering, ensuring the system adapted its behavior to the specific needs of the query type.

3. Key Contributions

1. R2RAG Architecture: A novel dynamic routing system that explicitly separates simple and complex query handling, optimizing resource usage while maximizing answer quality.
2. Resource-Efficient Design: Demonstrated that high-quality, dynamic RAG can be achieved on consumer hardware using compact, unquantized 4B parameter models, challenging the notion that large-scale RAG requires massive infrastructure.
3. Qualitative-Driven Optimization: The system was refined based on human feedback rather than just synthetic metrics, addressing the "distributional mismatch" often found in LLM-as-a-Judge evaluations.
4. Iterative Evidence Accumulation: The "Vanilla Agent" component introduces a robust mechanism for self-correcting search loops, where the system actively identifies knowledge gaps and reformulates queries until a coverage threshold is met.

4. Results

• Competition Performance: R2RAG won the Best Dynamic Evaluation award in the Open Source category of the NeurIPS 2025 MMU-RAG Competition.
• Effectiveness: The system excelled in the "RAG-Arena" (real-time dynamic) mode, outperforming static evaluation baselines.
• Trade-off Resolution: The routing mechanism successfully balanced the trade-off between the conciseness required for simple queries and the depth required for complex research tasks, a distinction that aggregate metrics often miss.

5. Significance

• Shift in Evaluation Paradigm: The paper highlights that live, dynamic evaluation and qualitative user feedback are essential for developing robust RAG systems, as static metrics do not fully capture reasoning depth or user preference.
• Accessibility: By proving that consumer-grade GPUs can power sophisticated, adaptive RAG agents, the work lowers the barrier to entry for deploying advanced AI research tools.
• Reproducibility: The authors released their code repository, promoting transparency and allowing the community to build upon their lightweight, dynamic architecture.
• Future Direction: The study suggests that future RAG systems should prioritize adaptive planning and human-aligned evaluation over brute-force scaling of model parameters.