Securing LLM-as-a-Service for Small Businesses: An Industry Case Study of a Distributed Chatbot Deployment Platform

Technical Summary: Securing LLM-as-a-Service for Small Businesses

Problem Statement
Small businesses face significant barriers to adopting Large Language Model (LLM)-based question-answering systems, particularly those utilizing Retrieval-Augmented Generation (RAG). These barriers include prohibitive infrastructure costs, engineering complexity, and security risks. Specifically, RAG-based systems introduce new attack surfaces where malicious instructions embedded in user queries or retrieved documents (prompt injection) can bypass system constraints, leading to data leakage or policy violations. Existing solutions often rely on expensive, centralized cloud GPU services or require enterprise-scale engineering teams and model retraining, which are inaccessible to resource-constrained small organizations. Furthermore, multi-tenant deployments risk cross-tenant data leakage and lack robust isolation mechanisms suitable for heterogeneous, low-cost hardware environments.

Methodology
The authors present an industry case study of an open-source, multi-tenant platform designed to deploy customized LLM support chatbots via a no-code workflow. The methodology encompasses three core architectural and security components:

1. Distributed Infrastructure: The platform is built on lightweight, high-availability k3s (Kubernetes) clusters spanning heterogeneous, low-cost machines. These clusters are interconnected via an encrypted overlay network with 60–200 ms latency. This design enables cost-efficient resource pooling while enforcing container-based isolation and per-tenant data access controls, avoiding reliance on uniform, high-performance cloud infrastructure.
2. Layered Security for Prompt Injection: To mitigate prompt injection attacks without model retraining, the platform implements a two-layer defense strategy:
   • Guard Prompts: Static, system-level prompts embedded directly into the LLM's context. These enforce behavioral constraints (e.g., prohibiting role switching, instruction overriding, or data exfiltration) and are model-agnostic.
   • GenTel-Shield Integration: A pre-trained, model-agnostic detection model deployed as a pre-generation filter. It classifies user queries and retrieved content to identify and block malicious prompt injection attempts before they reach the LLM.
3. Data Handling: The platform incorporates a security-aware RAG workflow where tenant documents undergo PII screening and de-identification prior to indexing.

Key Contributions

• Open-Source Platform: The development of a multi-tenant LLM deployment platform tailored for small businesses, operating on distributed k3s clusters with an overlay network to function under realistic cost and operational constraints.
• Integration of Security Mechanisms: A demonstration of how platform-level security, including container isolation and layered prompt injection defenses (Guard Prompts + GenTel-Shield), can be integrated into RAG systems without requiring enterprise infrastructure or model fine-tuning.
• Empirical Evaluation: A real-world case study involving an e-commerce client (All Table Sports Australia) that evaluates both the security effectiveness and operational efficiency of the proposed deployment.

Results
The evaluation was conducted using a dataset of 250 benign queries and 250 adversarial prompt injection attempts across three LLMs (GPT-4.1-mini, GPT-4.1, and Ministral-3B).

• Security Effectiveness:
  • Pure LLM: Exhibited extremely low recall (0.40–1.20 across models), indicating that base LLMs rarely proactively intercept prompt injection attacks.
  • Guard Prompts: Achieved near-perfect defensive performance with recall rates of 99.6–100% and F1 scores approaching 100%. However, this relies on careful, scenario-specific prompt engineering.
  • GenTel-Shield: Provided strong but incomplete protection (F1 ~89–90%) with high precision (99.51%) but moderate recall (81.6%).
  • Combined Strategy: The combination of Guard Prompts and GenTel-Shield yielded the most robust results, achieving 100% recall and ~99.8% F1 scores across all models, demonstrating the complementary nature of rule-based constraints and learned detection.
• Performance and Latency:
  • Contrary to the expectation that containerization adds overhead, the k3s-based private cloud deployment demonstrated lower end-to-end inference latency compared to bare-metal deployments.
  • Latency reductions ranged from approximately 28% (GPT-4.1-mini) to over 60% (Ministral-3B) compared to bare-metal execution.
  • The inclusion of Guard Prompts introduced only a modest, stable latency overhead, confirming that the private cloud architecture supports additional security mechanisms efficiently.

Significance and Claims
The paper claims that secure and efficient LLM-based chatbot services can be achieved for small businesses under realistic constraints, challenging the notion that production-grade RAG systems require hyperscale cloud environments. The findings provide actionable guidance for practitioners on balancing cost, security, and usability. Specifically, the study suggests that:

1. Relying solely on intrinsic LLM safety mechanisms is ineffective against prompt injection in RAG systems.
2. A layered defense combining static guard prompts with automated detection offers a practical balance between security effectiveness and long-term deployability.
3. Lightweight, containerized private cloud architectures can deliver inference performance comparable to or better than bare-metal setups while providing necessary multi-tenant isolation and security features.

The authors emphasize that these solutions are designed for constrained, domain-bounded question-answering rather than high-stakes autonomous decision-making, aligning with the operational realities of small enterprises.