From Prompt to Service: An SLM-Based Agent Orchestration Gateway for AI-Driven Virtual Worlds

Technical Summary: An SLM-Based Agent Orchestration Gateway for AI-Driven Virtual Worlds

Problem Statement
As generative AI capabilities expand, AI-driven virtual worlds face a significant architectural challenge: users interact through multimodal interfaces, but their requests demand fundamentally different backend models and computational resources (e.g., conversational dialogue, translation, domain-specific retrieval, or 3D asset generation). Directly embedding these diverse capabilities into virtual world clients reduces extensibility, complicates maintenance, and limits the ability to coordinate services distributed across edge and cloud infrastructure. Current approaches often lack a runtime coordination layer capable of interpreting user intent and routing requests to heterogeneous AI services without tightly coupling the client to specific endpoints, payload formats, or deployment locations.

Methodology
The authors propose and evaluate an SLM-based Agent Orchestration Gateway, a lightweight runtime coordination mechanism designed to decouple virtual world clients from heterogeneous AI backends. The system operates as follows:

1. Architecture: The gateway is implemented as a Python-based FastAPI service running on an edge device (NVIDIA Jetson Orin NX 8GB). It exposes a single REST endpoint to the virtual world client.
2. Intent Routing: Upon receiving a user prompt, the gateway forwards it to an Intent Router powered by a Small Language Model (SLM). The router classifies the semantic intent and returns a structured JSON decision (selected route, intent label, confidence score, and normalized query).
3. Service Registry & Validation: A configurable registry maintains the catalogue of available services (e.g., RAG, fine-tuned LLMs, text-to-3D). The gateway validates the router's decision against this registry before invoking the appropriate backend.
4. Evaluation Setup: The system was tested within the InterwovenXR virtual museum testbed, focusing on a scenario involving the Panagia Aggeloktisti Church. The evaluation addressed two Research Questions (RQs):
   • RQ1: Can compact SLMs serve as reliable intent routers on edge hardware?
   • RQ2: Does a layered configuration (separating intent recognition from response generation) offer practical advantages over a single-model setup?
5. Experimental Design:
   • Stage 1: Evaluated routing accuracy, invalid-output rates, and latency using a 500-prompt dataset across five service routes (Conversational, Language Adaptation, Domain Expert, Knowledge Retrieval, Asset Generation). Various SLMs (including Qwen2.5 and SmolLM2 families) were tested, with sub-billion-parameter models undergoing task-specific fine-tuning via 4-bit QLoRA.
   • Stage 2: Evaluated end-to-end latency for a layered configuration where a fine-tuned sub-billion model handled routing, and a larger compact SLM handled conversational response generation locally.

Key Results

• Routing Accuracy: Base sub-billion models (e.g., Qwen2.5-0.5B, SmolLM2-360M) performed poorly without adaptation (16–24% accuracy). However, task-specific fine-tuning significantly improved performance. The fine-tuned Qwen2.5-0.5B-FT achieved 83.00% routing accuracy with an average latency of 392.0 ms, a substantial improvement over its base model while maintaining low latency.
• Invalid Outputs: Fine-tuning also reduced invalid JSON output rates, though SmolLM2-360M-FT still exhibited a higher rate (22.60%) compared to Qwen2.5-0.5B-FT (3.20%).
• Layered Configuration: The layered approach (fine-tuned router + larger conversational model) proved feasible on mid-range edge hardware. All four tested pairings achieved end-to-end average latencies below 1.6 seconds (median < 1.5s), meeting the 2-second threshold for interactive use.
• Efficiency: The layered configuration was found to be faster than using a single larger model for both routing and generation. For instance, the SmolLM2-360M-FT + Qwen2.5-1.5B pairing achieved an average latency of 1306.06 ms, demonstrating that separating the decision stage reduces computational cost without sacrificing response quality for conversational tasks.

Significance and Contributions
The paper claims three primary contributions:

1. Architectural Pattern: It introduces a decoupled, intent-driven service routing architecture that allows virtual agents to act as unified access points to distributed generative AI services, enabling extensibility without client-side modifications.
2. Viability of Fine-Tuned SLMs: It demonstrates that compact, sub-billion-parameter models, when fine-tuned for specific routing tasks, can serve as reliable, low-latency intent routers on edge hardware, overcoming their limitations in general instruction following.
3. Layered Orchestration Strategy: It provides empirical evidence that a layered configuration—using a very small model for fast intent recognition and a larger compact model for response generation—is more efficient and practical for edge-supported virtual worlds than delegating both tasks to a single model.

Limitations and Future Work
The authors note that the evaluation was conducted within a controlled dataset and a single virtual museum scenario, limiting generalizability to other domains. The study measured routing accuracy and latency but did not assess the perceptual quality of generated responses through user studies. Additionally, the fine-tuned models are specific to the defined routing schema, requiring retraining if service definitions change. Future work aims to evaluate smaller architectures (<500M parameters), expand the dataset to include multi-turn dialogues and ambiguous prompts, and conduct full interactive user studies to measure engagement and immersion.