Toward Safe and Human-Aligned Game Conversational Recommendation via Multi-Agent Decomposition

1. Problem Statement

The paper addresses the limitations of existing Conversational Recommender Systems (CRS) when applied to the video game domain, specifically platforms like Roblox. While CRS has succeeded in static domains like movies, game recommendation presents three unique challenges:

1. Complex User Constraints: Unlike movies, game preferences are driven by interactive factors (mechanics, skill level, hardware compatibility, social modes) rather than just content themes.
2. Knowledge Recency Gap: Game catalogs evolve rapidly due to user-generated content. Large Language Models (LLMs) suffer from training data cutoffs, leading to poor recognition of niche or newly released titles and high rates of hallucination.
3. Safety and Transparency Risks: The interactive nature of games increases exposure to adversarial prompts (e.g., "jailbreaks" asking for harmful content) and policy violations. Existing CRS often lack robust safety layers and fail to provide transparent explanations, eroding user trust.

2. Methodology: The MATCHA Framework

The authors propose MATCHA (Multi-Agent System Collaboration for Trustworthy Conversational Recommendations), a modular multi-agent framework designed to decompose the recommendation pipeline into specialized agents.

Core Architecture

The system processes user queries through a pipeline of distinct agents:

• Risk Control Module (Safety):
  • Jailbreak Prevention Agent: Uses a three-pronged approach: (1) RA-LLM (random token dropping) to detect sensitivity changes, (2) Chain-of-Thought (CoT) intent detection to reason about semantic intent, and (3) Crafted Policies for standardized fallback behaviors.
  • Dangerous Content Detection Agent: A secondary filter for both input queries and output recommendations to flag harmful content.
• Candidate Generation Module:
  • Tool-Augmented Retrieval: Instead of relying solely on LLM parametric knowledge, this agent utilizes over 10 specialized tools (APIs) to access real-time data. These tools filter by platform, genre, trends, user feedback, and device compatibility.
  • Intent Parsing: An LLM analyzes the user query to extract specific constraints (e.g., "solo play," "RPG") to refine the candidate pool.
• Ranking and Reflection Module:
  • Multi-LLM Ranking Agent: Employs a two-tier collaboration where multiple LLMs (e.g., GPT-4o, Gemini) independently rank candidates across five metrics (popularity, preference match, history similarity, genre alignment, age suitability). Scores are combined via weighted averages to leverage diverse model strengths.
  • Reflection Agent: Performs a "reranking" phase using detailed game profiles (metadata) and user context. It re-evaluates the top candidates to ensure alignment, trading off some diversity for higher accuracy.
• Explainability Module:
  • Explanation Agent: Generates user-centric rationales for recommendations based on four dimensions: Category Preferences, Similarity to past choices, Demographics (age appropriateness), and Popularity/Novelty. It synthesizes these into a coherent summary to build trust.

3. Key Contributions

1. Novel Multi-Agent Architecture: A specialized framework for game CRS that coordinates intent parsing, tool-augmented retrieval, multi-LLM ranking, reflection, and safety control.
2. Comprehensive Evaluation: Extensive testing on real-world user data (Roblox) and adversarial benchmarks, demonstrating superiority over single-agent and traditional multi-agent baselines.
3. Safety and Transparency Integration: A dedicated focus on mitigating adversarial attacks (jailbreaks) and providing interpretable explanations, addressing a gap in existing CRS literature.
4. Practical Deployment Insights: The system was implemented and tested internally, providing real-world data on latency, cost, and user alignment.

4. Experimental Results

The system was evaluated on a dataset of 553 real user requests (Roblox) and adversarial benchmarks (WildJailbreak, DAN).

• Performance Metrics:
  • Relevance: Improved Hit@5 by 20% compared to baselines.
  • Novelty: Reduced popularity bias by 24% (lower Pop50@k), indicating better coverage of long-tail content.
  • Factuality: Achieved a near-perfect 0.99 score, significantly outperforming base LLMs which hallucinate frequently in the gaming domain.
  • Safety: Achieved 97.9% adversarial defense on jailbreak datasets, significantly outperforming GPT-4o (70.13%) and MACRec (74.83%).
  • Explanation Quality: Scored 4.2/5 by a virtual judge and 3.97/5 by human domain experts, confirming high alignment with human expectations.
• Ablation Studies:
  • Removing the Jailbreak Prevention Agent dropped defense success from 97.85% to 88.5%.
  • Removing Multi-LLM collaboration and Reflection significantly reduced relevance and accuracy, proving the necessity of the multi-agent approach.
• Generalization: When adapted to the ReDial (movie) dataset, MATCHA maintained strong performance (Recall@10: 0.182), demonstrating its flexibility across domains.

5. Significance and Impact

• Domain Adaptation: The paper establishes that game recommendation requires a fundamentally different approach than movie/book recommendation, necessitating tool-augmented retrieval and dynamic constraint handling.
• Safety First: It provides a blueprint for integrating robust safety mechanisms (jailbreak detection, content filtering) directly into the recommendation pipeline, crucial for platforms with user-generated content.
• Trust and Explainability: By explicitly generating multi-dimensional explanations, MATCHA bridges the "black box" gap, fostering user trust in automated recommendations.
• Scalability: Despite using multiple LLMs and tools, the system maintains low latency (approx. 1.32s per turn) and low cost, proving that complex multi-agent systems can be deployed efficiently in real-world settings.

In conclusion, MATCHA represents a significant step forward in making conversational AI safe, accurate, and trustworthy for high-engagement, dynamic domains like gaming.