EventChat: Implementation and user-centric evaluation of a large language model-driven conversational recommender system for exploring leisure events in an SME context

This paper presents the design, implementation, and user-centric evaluation of an LLM-driven conversational recommender system for SMEs, revealing high recommendation accuracy but highlighting significant challenges in latency and cost—particularly when using advanced LLMs as rankers—that must be addressed to ensure business viability.

The Gist

The Big Picture: A Smart Travel Agent for a Small Business

Imagine you run a small, local travel agency (a "Small and Medium Enterprise" or SME). You have a massive catalog of thousands of events—concerts, theater shows, pool parties, and food festivals.

Traditionally, your customers have to use a boring filter menu: "Show me concerts under $50 happening next week." It's rigid, and if they don't know exactly what they want, they get overwhelmed.

Enter EventChat. This is a new kind of digital assistant powered by Large Language Models (LLMs)—the same "brain" technology behind tools like ChatGPT. Instead of clicking filters, a customer can just chat with the app: "I'm in the mood for something spooky but cheap, maybe this weekend." The AI understands the vibe, checks your database, and suggests the perfect haunted house tour.

The paper is a report card on how a real startup tried to build this system, how it performed in the real world, and what they learned about the costs and headaches of using AI for small businesses.

---

1. The Goal: Why Build This?

The startup wanted two things:

1. Better Customer Experience: Make finding fun things to do easier and more fun.
2. Fix Bad Data: Their event list was built by "web scrapers" (robots that copy-paste info from other websites). Sometimes these robots miss details (like the price or the exact date). A human-like chatbot can read the messy text descriptions and figure out the details better than a rigid search bar.

2. The Design: Building the "Robot" on a Budget

The researchers didn't build a super-complex, expensive robot. They built a pragmatic, budget-friendly version suitable for a small company.

• The Brain: They used ChatGPT (via Microsoft Azure) because it was the smartest tool available at the time and they got it for free through a startup grant.
• The Strategy (Prompt-Based Learning): Instead of teaching the AI from scratch (which requires huge amounts of data and money, like sending a student to a 4-year university), they just gave it a set of instructions (a "prompt"). Think of it like hiring a very smart intern and giving them a detailed cheat sheet on how to do the job, rather than training them for years.
• The Workflow (Stage-Based): They didn't let the AI run wild. They built a "conveyor belt" system with five fixed steps:
  1. Chat: Just talk.
  2. Refusal: Say "no" if the question is weird.
  3. Search: Look for specific things.
  4. Recommend: Suggest things based on what you liked before.
  5. Inquiry: Answer specific questions about an event.
  • Analogy: Imagine a restaurant kitchen. Instead of letting the chef (the AI) wander around the store to find ingredients (which takes too long and costs too much), the chef stays at the station and follows a strict recipe card. It's faster and cheaper.

3. The Results: The Good, The Bad, and The Expensive

They tested this with real people in a German city. Here is what happened:

✅ The Good News

• It Works: Most people (85.5%) felt the recommendations were accurate.
• Low Effort: Users felt it was easy to find what they wanted.
• The "ResQue" Model: The researchers updated a famous survey model (called ResQue) to measure how people feel about AI chatbots. They added new questions about how "consistent" and "coherent" the chat felt. This new survey tool is now a blueprint for other researchers.

❌ The Bad News (The "Gotchas")

• The "Latency" Problem (Waiting Time): The system was slow. On average, it took 5.7 seconds just to reply to a single message.
  • Analogy: Imagine ordering a coffee, and the barista takes 6 seconds just to say "One coffee coming up!" before actually making it. It feels sluggish.
• The Cost Problem: Every time the AI thought, it cost money. The median cost was $0.04 per interaction.
  • Analogy: If you have 1,000 customers a day, that's $40 a day just for the AI to think. For a small business with thin profit margins, this adds up fast.
• The "Hallucination" Problem: Sometimes the AI made things up. It might suggest a concert that didn't exist or ignore the price limit you set.
  • Analogy: It's like a tour guide who confidently tells you a museum is open on Sundays, but they just made that up because they forgot to check the schedule.
• The "Context" Problem: The AI sometimes forgot details. If you said "I want something under $20," it might later suggest a $50 event because it got distracted by the long list of event descriptions.

4. Key Lessons for Small Businesses

The paper concludes with five "Golden Rules" for small companies trying to use AI:

1. Keep it Simple (Stage-Based): Don't let the AI be a free-roaming "agent" that tries to do everything. Give it a fixed list of tasks. It's cheaper and more stable.
2. Instructions Aren't Everything: Just giving the AI a prompt isn't enough if the task is too complex. Sometimes you need to "train" it (fine-tune) or give it better data, but that costs more.
3. Guardrails are Essential: You need safety nets to stop the AI from lying (hallucinating) or getting hacked.
4. Listen to How People Talk: Users ignored the "Time Button" on the screen and just typed their dates in the chat. Design your app to listen to the chat, not just the buttons.
5. The "Re-Ranking" Tax: The most expensive part of the system was the AI trying to pick the best 10 events out of 100. Using a super-smart AI to do this is like hiring a Nobel Prize-winning chef to chop onions. It works, but it's too expensive for a small business.

The Bottom Line

EventChat proves that small businesses can build cool, AI-powered chatbots to help customers find events. It's a powerful tool that feels like magic.

However, it's not a "set it and forget it" solution. It's like driving a high-performance sports car: it's fast and fun, but it burns a lot of gas (money), needs a skilled driver (technical know-how), and if you aren't careful, it might stall (latency) or take a wrong turn (hallucinations).

For a small business to succeed with this, they have to carefully balance the wow factor for the customer against the wallet factor for the business.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) offer transformative potential for Conversational Recommender Systems (CRS), existing research predominantly focuses on technical frameworks in controlled settings or state-of-the-art algorithmic performance. There is a significant gap in understanding:

• SME Constraints: How Small and Medium-sized Enterprises (SMEs), which face strict resource, budget, and skill constraints, can practically implement LLM-driven CRS.
• Real-World Trade-offs: The operational realities of latency, cost, and data quality in production environments versus theoretical capabilities.
• Evaluation Frameworks: The lack of a validated, user-centric evaluation model specifically adapted for LLM-driven CRS that combines subjective user feedback with objective system metrics.

2. Methodology

The authors conducted a field study involving the design, implementation, and evaluation of EventChat, a CRS deployed by a startup in the leisure industry (German market).

A. System Design (EventChat)

• Architecture: A stage-based pipeline (rather than an agentic loop) to ensure stability and cost control. The system operates in turns, where the LLM selects one of five fixed actions: Chat, Refusal, Search, Recommendation, or Targeted Inquiry.
• LLM & Learning Strategy: Utilized ChatGPT (via Azure OpenAI) with prompt-based learning (In-Context Learning and Few-Shot Chain-of-Thought). The authors deliberately avoided fine-tuning to minimize data requirements and engineering complexity for the SME.
• Retrieval-Augmented Generation (RAG):
  • Search/Recommendation: Uses existing retrieval infrastructure (Amazon Personalize/Vector Store) to generate a candidate slate.
  • Re-ranking: A critical step where ChatGPT acts as a ranker, processing a textual "digest" of up to 10 candidate events in a single prompt to filter for relevance.
  • Targeted Inquiry: Uses structured item digests to answer specific questions, avoiding complex SQL generation by the LLM (which failed in testing due to database complexity).
• Frontend: A hybrid interface (Flutter-based) combining a chat window with static buttons for time selection and category filtering. It includes a "visibility detector" that feeds the last three viewed event cards back into the LLM context.

B. Evaluation Approach

The study employed a dual-method evaluation:

1. Subjective (User-Centric): A field survey with 83 participants using a revised, short-form ResQue model. This model was adapted to include conversational quality dimensions: Consistency, Coherence, and Input Processing Performance.
2. Objective (System Metrics): Analysis of server logs capturing latency (response time), token usage (cost proxy), and interaction success rates.

3. Key Contributions

1. EventChat Implementation: A resource-conscious, end-to-end LLM-driven CRS designed specifically for SME constraints, demonstrating that complex systems can be built without fine-tuning or massive training datasets.
2. Revised ResQue Model: An updated theoretical framework for evaluating LLM-driven CRS that integrates conversational quality metrics (Consistency, Coherence) and validates their predictive power on user satisfaction and future use.
3. Empirical Field Data: One of the first studies to provide real-world metrics (latency, cost, token consumption) alongside user perceptions, moving beyond simulated benchmarks.
4. Design Principles for SMEs: Identification of five actionable design insights regarding architecture, prompt engineering, and interface design for resource-constrained environments.

4. Key Results

A. Performance Metrics

• Latency: Median latency was 5.7 seconds per message and 13.7 seconds per session. The re-ranking stage was the primary bottleneck, consuming ~4.0 seconds of the total time.
• Cost: The median cost per interaction was $0.04 (based on August 2023 token pricing). The re-ranking phase consumed the most tokens (median 23,408 tokens per session), making it the most expensive component.
• Token Usage: High token consumption was driven by the need to include full event digests in the prompt for the re-ranking step.

B. User Evaluation (Subjective)

• Accuracy: 85.5% of users rated recommendation accuracy as neutral or good. However, 15% rated it poorly, often due to missing data from the startup's web-scraping sources.
• Success Rate: 83.1% of requests were successfully fulfilled. Failures were primarily caused by:
  • Lack of relevant events in the database (data quality issues).
  • LLM failure to adhere to language constraints (switching from English to German).
  • Inability to extract time preferences from chat when users ignored UI buttons.
• Model Validation: Path analysis confirmed that Recommendation Accuracy and Consistency significantly drive Perceived Usefulness, which in turn drives Confidence, Satisfaction, and Future Use.

C. Critical Findings & Limitations

• Prompt-Based Limitations: Relying solely on prompts led to occasional hallucinations (suggesting non-existent events) and context blindness (ignoring price or time constraints).
• Cost-Quality Trade-off: Using a large LLM (ChatGPT) as a re-ranker is economically prohibitive at scale for SMEs.
• Interface Mismatch: Users often ignored static UI buttons for time selection, preferring to type in the chat, which the system struggled to parse correctly without explicit filters.

5. Significance and Implications

Technical Significance

• Architecture Choice: The study argues that for SMEs, stage-based architectures are superior to agentic pipelines due to better cost predictability and lower latency variance.
• Re-ranking Bottleneck: It highlights that using general-purpose LLMs for slate-level re-ranking is a major cost driver, suggesting a need for smaller, fine-tuned models or alternative ranking strategies for production.
• Guardrails: The paper emphasizes the necessity of architectural guardrails (e.g., validating item IDs against the database) to mitigate hallucinations in production.

Managerial Significance

• Feasibility: LLM-driven CRS are feasible for SMEs but require careful cost-benefit planning. The current technology is most viable in high-margin scenarios where the cost per interaction ($0.04) is acceptable.
• Strategic Trade-offs: SMEs must balance the desire for anthropomorphism and flexibility against the realities of latency and operational costs.
• Data Quality: The system's performance is heavily dependent on the underlying data quality; LLMs cannot fully compensate for structural data gaps (e.g., missing price fields) without specific engineering workarounds.

Theoretical Significance

• The study validates that core constructs from traditional RS evaluation (like Perceived Usefulness) remain relevant in the Gen-AI era, but conversational-specific constructs (Consistency, Input Processing) are now critical predictors of user satisfaction.
• It provides a roadmap for future research to move from algorithmic benchmarking to holistic, user-centric evaluation in real-world business contexts.

In conclusion, EventChat demonstrates that while LLM-driven CRS can be successfully deployed by SMEs to enhance user experience, the path to widespread adoption is currently blocked by latency, operational costs, and the limitations of prompt-only approaches in complex, data-sparse environments. Future success depends on optimizing the cost-latency-quality triangle through architectural innovations and potentially hybrid model approaches.