Technical Summary: STREAM – Multi-Tier LLM Inference Middleware with Dual-Channel HPC Token Streaming

Problem Statement

Researchers and practitioners face a fragmented landscape when deploying Large Language Models (LLMs), forced to choose between three imperfect options:

1. Local Inference: Free and private but limited by user hardware constraints (smaller models, limited context windows).
2. Institutional HPC: Offers powerful GPU resources at no marginal cost and keeps data within institutional boundaries, but operates behind firewalls that block standard streaming approaches and relies on batch execution models that lack mechanisms for incremental output delivery.
3. Commercial Cloud APIs: Provide frontier-model quality on demand but impose significant per-token costs and data retention policies unsuitable for sensitive research data.

No existing system unifies these three tiers, leading to unnecessary cloud spending, underutilized institutional GPU resources, and fragmented workflows. Furthermore, standard streaming architectures fail in HPC environments because compute nodes sit behind institutional firewalls that block all inbound connections, and client machines are often behind NAT, making them unreachable without VPNs or firewall rule changes.

Methodology and System Architecture

STREAM (Smart Tiered Routing Engine for AI Models) acts as a middleware layer unifying local, HPC, and cloud inference into a single tiered service. The system processes every query through a standardized pipeline:

1. Three-Tier Routing Architecture

Queries are classified by a Complexity Judge (a local LLM, specifically Llama 3.2 3B) into three tiers:

• LOW: Routed to local models (e.g., Llama 3.2 3B, Gemma 3 4B-VL) running on user devices or co-located servers via Ollama.
• MEDIUM: Routed to institutional HPC resources (e.g., Qwen 2.5-VL-72B-AWQ on H100 NVL GPUs).
• HIGH: Routed to commercial cloud APIs (e.g., via OpenRouter).
  The system includes a fallback mechanism where queries escalate or descend tiers based on availability and error states, ensuring continuity.

2. Dual-Channel HPC Token Streaming

To solve the HPC streaming problem, STREAM introduces a novel architecture that separates the Control Plane from the Data Plane:

• Control Plane (Globus Compute): Handles federated authentication and job dispatch. It submits a task containing the query and metadata but does not carry token payloads.
• Data Plane (WebSocket Relay): A cloud-based relay server acts as a rendezvous point.
  • Producer: The HPC worker (running inside a Globus Compute task) connects outbound to the relay, reads the vLLM HTTP SSE stream token-by-token, and forwards them.
  • Consumer: The client (or proxy) connects outbound to the same unique UUID channel on the relay to receive tokens.
• Firewall Traversal: Since both endpoints initiate outbound connections (wss:// on port 443), the architecture traverses institutional firewalls and NAT without requiring inbound ports, VPNs, or firewall rule changes.
• Security: The system employs end-to-end AES-256-GCM encryption. Encryption keys and channel secrets are pre-loaded into the worker environment and never transmitted as task arguments, ensuring the relay operator cannot read token payloads.

3. Tier-Aware Context Summarization

To prevent long conversation histories from forcing simple queries onto expensive tiers due to context window limits, STREAM implements differential compression. When a conversation approaches 80% of a tier's context limit, older messages are compressed into a summary (using the free local model) while recent turns are kept verbatim. This allows simple queries to remain on the local tier even after 40+ turns.

4. HPC-as-API Proxy Mode

STREAM exposes the HPC tier as an OpenAI-compatible endpoint (/v1/chat/completions). This allows external applications to access institutional HPC inference without requiring users to manage Globus credentials or implement the relay protocol directly. This mode is practical only because the dual-channel streaming reduces latency to sub-second levels.

Key Results

The system was evaluated on a 1,200-query benchmark spanning ten domains (HPC, mathematics, physics, etc.) and measured across local, HPC, and cloud tiers.

• Routing Accuracy: The Llama 3.2 3B complexity judge achieved 49.0% overall accuracy (1.5× above random chance) and 85.1% free-tier retention, successfully keeping 85% of queries on free local or HPC tiers. The primary limitation was low recall for "LOW" complexity queries, which were often over-routed to "MEDIUM" (a conservative error as both are free).
• Latency (TTFT):
  • Local: 0.26 s median Time-to-First-Token (TTFT).
  • HPC (Streaming): 0.54 s median TTFT. This represents a 21.1× improvement over the batch fallback mode (11.40 s), where TTFT equals total generation time.
  • Cloud: 1.68 s median TTFT.
• Throughput: Both HPC streaming and batch modes achieved identical generation throughput (26.9 tok/s), confirming the relay adds no per-token overhead.
• Context Summarization: In synthetic tests, summarization prevented forced tier upgrades for simple queries ("What is 2+2?") even when raw context exceeded 41K tokens (turn 40), keeping the query on the local tier.

Significance and Claims

The paper claims STREAM is the first system to unify local, institutional HPC, and cloud LLM inference with automatic complexity-based routing. Its primary significance lies in solving the HPC streaming problem, enabling real-time token delivery from behind institutional firewalls without VPNs or firewall modifications.

By reducing HPC TTFT from 11.40s to 0.54s, STREAM makes the "HPC-as-API" deployment pattern practical for interactive applications, a use case previously impossible due to batch latency. The system demonstrates that institutional GPU resources can be made as accessible as cloud APIs while maintaining data privacy and reducing costs. The authors note that STREAM is currently a research prototype; future work involves replacing the LLM-based judge with a fine-tuned text classifier and gathering real-user routing traces. The core components (relay and proxy) are released as domain-agnostic PyPI libraries (streamrelay and hpc-as-api).