MATRAG: Multi-Agent Transparent Retrieval-Augmented Generation for Explainable Recommendations

1. Problem Statement

The paper addresses three critical limitations in current Large Language Model (LLM)-based recommendation systems:

• Opacity: Existing LLM recommenders often function as "black boxes," generating suggestions without articulating the reasoning behind them, which erodes user trust.
• Hallucination & Lack of Grounding: LLMs frequently hallucinate facts or fail to incorporate up-to-date, domain-specific information. Standard Retrieval-Augmented Generation (RAG) often retrieves isolated text chunks, ignoring the rich relational structure of Knowledge Graphs (KGs).
• Coordination Challenges: Single-agent LLM systems struggle to decompose complex recommendation tasks, maintain coherent reasoning across multiple steps, and balance diverse signals (collaborative filtering vs. content-based) from heterogeneous sources.

2. Methodology: The MATRAG Framework

MATRAG (Multi-Agent Transparent Retrieval-Augmented Generation) is a novel framework that unifies multi-agent collaboration with knowledge graph-augmented retrieval. It orchestrates four specialized LLM-based agents via a central Orchestrator:

A. Core Agents

1. User Modeling Agent:
   • Constructs dynamic, multi-faceted user profiles ($P_u$) by processing behavioral signals (clicks, ratings), textual feedback (reviews), and contextual factors.
   • Distinguishes between explicit preferences, implicit behavioral inferences, and temporal evolution.
2. Item Analysis Agent:
   • Retrieves structured knowledge from a domain-specific Knowledge Graph (KG).
   • Performs entity linking, $k$-hop subgraph extraction, and relation filtering.
   • Uses a two-stage retrieval process: dense vector retrieval followed by LLM-based re-ranking to select the most relevant knowledge subgraphs ($\hat{S}_i$) based on user context.
3. Reasoning Agent:
   • Synthesizes signals from the User Modeling and Item Analysis agents.
   • Integrates Collaborative Filtering (CF) scores, Content-Based (CB) scores, and LLM-based preference predictions.
   • Generates a structured Reasoning Chain ($C$) that pairs reasoning steps with specific evidence from the retrieved KG, ensuring the decision process is traceable.
4. Explanation Agent:
   • Transforms the reasoning chain into natural language explanations.
   • Operates in three modes: Concise, Detailed, and Comparative.
   • Crucially, it is instructed to ground explanations strictly in the retrieved evidence and reasoning chain to prevent hallucination.

B. Transparency Scoring Module

A unique component of MATRAG is a quantitative scoring mechanism that evaluates explanation quality along three dimensions before final output:

1. Faithfulness: Measures if the explanation is entailed by the retrieved evidence (using NLI models).
2. Coherence: Evaluates logical consistency and flow (using an LLM-as-a-judge).
3. Personalization: Measures alignment between the explanation and the user's profile embeddings.

• The module outputs a composite Transparency Score, which can be used for automated filtering or Reinforcement Learning from Human Feedback (RLHF) to refine the system.

3. Key Contributions

• Framework Innovation: Introduction of MATRAG, the first framework to combine multi-agent collaboration, KG-augmented retrieval, and explicit explainability in a unified architecture.
• Transparency Mechanism: Proposal of a novel Transparency Scoring Module that quantifies explanation faithfulness, coherence, and personalization, enabling iterative quality improvement.
• State-of-the-Art Performance: Demonstration of superior performance across recommendation accuracy and explanation quality compared to traditional, LLM-based, and agent-based baselines.
• Human Validation: Empirical evidence from domain experts confirming the trustworthiness and helpfulness of the generated explanations.

4. Experimental Results

The framework was evaluated on three benchmark datasets: Amazon Reviews, MovieLens-1M, and Yelp.

Recommendation Accuracy

MATRAG outperformed all baselines (including BPR, LightGCN, KGAT, InteRecAgent, and K-RagRec):

• Amazon: Improved Hit Rate (HR@10) by 12.7% and NDCG@10 by 15.3% over the best baseline.
• MovieLens: Achieved significant gains in HR and NDCG, validating the efficacy of KG-augmented reasoning.
• Yelp: Consistently achieved the highest scores across all metrics.

Explanation Quality

• Faithfulness: MATRAG achieved a 0.847 faithfulness score (vs. 0.751 for the next best, G-CRS), indicating explanations were highly grounded in retrieved facts.
• Coherence: Scored 4.42/5, demonstrating logical consistency.
• Human Evaluation: In a study with 12 domain experts, 87.4% of MATRAG's explanations were rated as "helpful" and "trustworthy" (score $\ge$ 4), significantly outperforming the next best method (62.1%).

Ablation Studies

• Removing the Reasoning Agent caused the largest drop in accuracy (-15.4% HR), confirming its central role.
• Removing the Transparency Scoring module slightly reduced accuracy, suggesting the feedback loop helps refine the reasoning process itself.
• A single-agent variant (one LLM doing all tasks) underperformed the multi-agent system by 14.0%, validating the need for specialization.

5. Significance and Implications

• Trust in AI: MATRAG provides a roadmap for building "trustworthy" recommenders by making the reasoning process explicit and verifiable, addressing the "black box" problem in high-stakes domains like finance and healthcare.
• Architecture Design: The paper demonstrates that decomposing complex tasks into specialized agents (User, Item, Reasoning, Explanation) yields better results than monolithic LLM prompting.
• Grounding: It proves that integrating Knowledge Graphs directly into the RAG pipeline is essential for preventing hallucinations in recommendation explanations.
• Future Directions: While current latency is higher due to multi-agent coordination, the authors suggest asynchronous execution and agent caching as viable solutions for production deployment.

In conclusion, MATRAG establishes a new benchmark for transparent, agentic recommendation systems, showing that high accuracy and high explainability are not mutually exclusive but can be achieved synergistically through structured multi-agent collaboration and knowledge grounding.