💬 NLP

ShopEase: A Generative AI-Based Multi-Agent Framework for Intelligent Enterprise Customer Support Using Hybrid Retrieval-Augmented Generation

This paper introduces ShopEase, a multi-agent generative AI framework for enterprise customer support that leverages local LLaMA 3.2 and hybrid retrieval, demonstrating through evaluation on 2,632 queries that dense FAISS retrieval outperforms sparse and reranked hybrid approaches in accuracy while avoiding the latency penalties associated with cross-encoder reranking.

Original authors: Aakash Kumar Tiwari, Somesh Kumar

Published 2026-09-15✓ Author reviewed ⓘ

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Aakash Kumar Tiwari, Somesh Kumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world of business, keeping a customer happy often means answering their questions quickly and accurately. When a person buys something online and then needs to know about a refund, a shipping delay, or a damaged item, they expect an immediate answer. For decades, companies have tried to automate this with chatbots, computer programs designed to chat like humans. These programs work well for simple, repeated questions, but they often stumble when a situation gets complicated. If a customer asks about a specific policy while also mentioning a past order or a unique problem, a basic program might get lost because it cannot remember the conversation or find the right rulebook. To solve this, researchers are now building systems that act more like a team of specialists rather than a single robot. These new systems use a method called retrieval-augmented generation, which is a fancy way of saying the computer looks up facts in a database before it speaks, ensuring its answer is based on real company rules rather than just guessing. They also use multiple "agents," or small programs that handle different jobs, such as one that remembers who the customer is, another that finds the right policy, and a third that decides if a human needs to step in.

A team of researchers at the Indian Institute of Technology Kharagpur has built a new system called ShopEase to test how well this team approach works for enterprise customer support. Their goal was to create a framework that could handle the full journey of a customer query, from the moment a question is asked to the final answer, while knowing when to pass the baton to a human worker. The system is designed with six distinct parts that work together. First, a guard checks the incoming question to make sure it is safe and clear. Then, an agent identifies what the customer actually wants. Other agents pull up the customer's personal history from a database and recall details from previous conversations. A central retrieval team then searches through thousands of policy documents to find the specific rules that apply to the situation. If the system feels the problem is too complex or risky, an escalation agent hands the case over to a human. Finally, a supervisor coordinates all these steps and ensures the final response is reviewed before it is sent back to the customer.

To see if this design actually works, the researchers tested it on a large set of 2,632 real customer questions that had been kept aside for this specific experiment. These questions covered six common categories: refunds, returns, shipping issues, cancellations, damaged products, and a group of questions that did not fit any clear category. The team wanted to see which method of finding the right policy document worked best. They tried six different search strategies. Some relied only on matching exact words, while others used a method that understood the meaning behind the words, even if the specific vocabulary was different. They also tested combinations of these methods and added a step where a second, more careful check was performed on the top results to see if it improved the final answer.

The results showed that understanding the meaning of the question was far more important than matching exact words. The system that relied solely on finding documents with similar meanings achieved the highest success rate, correctly identifying the right policy category for 85.37 percent of the questions. This was significantly better than the system that only looked for matching words, which got the answer right for only 55.74 percent of the cases. Interestingly, adding that extra, careful check step did not help. In fact, it made the system slower without making it more accurate. The researchers found that the time it took to get an answer increased noticeably when they added this extra layer of processing, yet the number of correct answers did not go up. This suggests that for this type of task, a fast, meaning-based search is more effective than a slower, multi-step process.

When the researchers looked closer at where the system succeeded and where it failed, a clear pattern emerged. The system was very good at handling questions about shipping, cancellations, and returns, often getting the right answer more than 90 percent of the time. However, it struggled significantly with questions that did not fit neatly into a known category. When a customer asked a vague question that did not match any specific policy, the system often guessed and assigned it to a known category anyway, rather than admitting it did not know. This was the main source of errors. The system also had trouble distinguishing between refunds and returns, as these two concepts often use similar language, leading to some confusion. Despite these hiccups, the study proved that a multi-agent system could successfully combine customer history, conversation memory, and policy retrieval into a single, coordinated workflow.

The researchers also tested what happened when they removed certain parts of the system to see how important each piece was. They found that removing the part that remembered the customer's personal details caused the biggest drop in how personalized the answers felt. This confirmed that knowing who the customer is matters just as much as knowing the rules. Removing the part that handles conversation history also hurt the quality of the answers, though not as much as losing the customer data. The part responsible for handing cases to humans had less impact on the quality of the automated answers, which makes sense because its main job is to step in when the machine cannot solve the problem, rather than to improve the machine's own performance.

In the end, the ShopEase system demonstrated that a team-based approach works well for enterprise support, provided the right tools are used. The study showed that dense retrieval, which understands the meaning of words, is superior to simple keyword matching for this kind of work. It also revealed that adding complex re-checking steps can slow things down without adding value. While the system is not perfect, especially when dealing with vague or out-of-scope questions, it offers a solid foundation for the future of customer service. The researchers suggest that future improvements should focus on better ways to detect when a question is outside the system's knowledge and on expanding the library of policies to cover more diverse scenarios. For now, this work provides a clear roadmap for building intelligent support systems that are fast, accurate, and ready to hand off to a human when the situation demands it.

Technical Summary: ShopEase – A Generative AI-Based Multi-Agent Framework for Intelligent Enterprise Customer Support

1. Problem Statement

Enterprise customer support systems face the challenge of accurately answering customer queries while integrating relevant customer context, policy information, and conversation history. Traditional chatbots often struggle when queries require specific policy retrieval, access to customer records (CRM), or human intervention for complex cases. Existing research tends to address these challenges in isolation: some focus on chatbot response generation, others on Retrieval-Augmented Generation (RAG) strategies, multi-agent coordination, or human-in-the-loop (HITL) mechanisms. There is a lack of a unified framework that combines these components with a comparative evaluation of different retrieval strategies (lexical, dense, hybrid, and reranked) within a single enterprise workflow. Furthermore, relying on fixed mappings between query intent and policy categories often fails when customer language diverges from policy wording.

2. Methodology

The paper proposes ShopEase, a Generative AI-based multi-agent framework designed to handle the complete enterprise customer support workflow. The system is implemented as a graph-based workflow in Python, utilizing a local LLaMA 3.2 model (via Ollama) for response generation and a Streamlit interface for user interaction.

Core Components:

Guardrail: Preprocesses and validates incoming customer queries.
Intent Agent: Identifies the primary intent of the query.
CRM Agent: Retrieves customer-specific data (e.g., order history, tier) from a SQLite database.
Memory Agent: Accesses relevant information from previous conversation turns.
Hybrid RAG Agent: Retrieves policy documents using a combination of dense and sparse retrieval methods.
Escalation Agent: Routes cases requiring human review to a human-in-the-loop stage.
Supervisor Agent: Coordinates the workflow, managing shared state between components.
Reflection Component: Reviews the generated response before final delivery.

Retrieval Strategy:
The system employs a Hybrid RAG approach for policy retrieval, combining:

Dense Retrieval: Using FAISS with nomic-embed-text embeddings for semantic similarity.
Sparse Retrieval: Using BM25 for lexical matching.
Fusion: Combining results via Reciprocal Rank Fusion (RRF).
Reranking: Evaluating the addition of a Cross-Encoder (ms-marco-MiniLM-L-6-v2) for re-ranking.

The policy category is determined directly from the retrieved documents rather than through a fixed intent-to-policy mapping.

Experimental Configurations:
Six retrieval configurations were evaluated on a held-out dataset of 2,632 queries across six categories (Refund, Return, Shipping, Cancellation, Damaged Product, Unknown):

BM25-only
FAISS-only
Fair RRF (equal weights)
Weighted RRF (optimized weights)
RRF + Cross-Encoder
Top-10 Hybrid + Cross-Encoder

3. Key Contributions

The paper outlines five primary contributions:

Framework Development: The creation of ShopEase, which integrates customer context, conversation memory, policy retrieval, response generation, and human escalation into a single multi-agent workflow.
Comparative Retrieval Analysis: A systematic evaluation of six distinct retrieval configurations (sparse, dense, hybrid, and reranked) under identical experimental conditions.
Dataset Evaluation: Testing on a substantial held-out dataset of 2,632 real-world enterprise customer queries.
Comprehensive Metrics: Analysis using accuracy, precision, recall, F1-scores, confusion matrices, latency measurements, and statistical significance testing (McNemar's test).
Ablation Study: An investigation into the impact of removing specific components (CRM, Memory, Escalation) on system performance, specifically regarding groundedness and personalization.

4. Results

The evaluation yielded the following key findings:

Accuracy: FAISS-only achieved the highest accuracy at 85.37% (2,247 correct predictions). Weighted RRF followed closely at 85.07%. BM25-only performed significantly lower at 55.74%.
Reranking Impact: The addition of Cross-Encoder reranking did not improve classification accuracy. "RRF + Cross-Encoder" achieved 83.24%, and "Top-10 Hybrid + Cross-Encoder" achieved 81.88%.
Latency: Reranking significantly increased processing time. Cross-Encoder configurations had mean latencies of ~0.14s and ~0.25s, compared to ~0.04s for FAISS-only and ~0.0005s for BM25-only.
Statistical Significance: McNemar's test showed no statistically significant difference between FAISS-only and Weighted RRF ( $p = 0.428$ ), though both were significantly better than Fair RRF and Cross-Encoder configurations.
Category Performance: The system performed strongly on Shipping (99.17% recall), Cancellation (96.89% recall), and Return (94.42% recall). The Unknown category remained a major source of error, with 0% recall, as the system tended to force these queries into known policy categories.
Error Patterns: The primary confusion occurred between "Refund" and "Return" due to overlapping terminology.
Ablation Findings: Removing the CRM component caused the most significant drop in personalization scores, highlighting the importance of customer-specific data. Removing the Escalation component had a minimal effect on response scores, as its primary role is workflow routing rather than content generation.

5. Significance and Claims

The paper claims that ShopEase demonstrates the viability of a unified multi-agent framework for enterprise support that effectively leverages customer context and conversation history. The study concludes that for the evaluated dataset, dense retrieval (FAISS) provides the best balance of accuracy and efficiency, outperforming sparse retrieval and hybrid approaches that include reranking.

The authors modestly claim that their work fills a gap by combining multiple support components (CRM, memory, RAG, HITL) and providing a direct, controlled comparison of retrieval strategies. They emphasize that while the system improves policy-based query handling, the current limitations include the handling of "Unknown" queries and the lack of long-term real-world deployment data. The paper does not claim that the system is universally superior in all contexts but rather that it offers a robust, coordinated workflow for specific enterprise policy retrieval tasks where dense semantic matching is advantageous.