Scalable Inference Architectures for Compound AI Systems: A Production Deployment Study

1. Problem Statement

Modern enterprise AI is shifting from standalone Large Language Models (LLMs) to Compound AI Systems—architectures that orchestrate multiple models, retrievers, and tools to solve complex tasks (e.g., Salesforce's Agentforce and ApexGuru). Deploying these systems in production presents unique infrastructure challenges that traditional model serving cannot address:

• Heterogeneous & Concurrent Invocations: A single user request triggers a "fan-out" of 3–5 distinct model calls (e.g., embedding, classification, LLM generation) with complex dependency graphs.
• Cascading Cold Starts: In static deployments, an idle period causes simultaneous cold starts across multiple dependent models. Because of pipeline dependencies, the user waits for the serial sum of these cold starts rather than the maximum, leading to massive latency spikes (e.g., ~180s).
• Resource Contention & Inefficiency: Static GPU pools suffer from resource contention when different model types compete for the same hardware. Furthermore, scaling based on total user requests leads to over-provisioning for conditional components (e.g., SQL executors) and under-provisioning for ubiquitous ones (e.g., embeddings).
• Rapid Iteration Bottlenecks: Traditional monolithic deployment pipelines cannot support the rapid, component-level A/B testing required to optimize multi-model systems.

2. Methodology & System Architecture

The authors designed a modular, platform-agnostic inference architecture deployed at Salesforce to support these workloads. The system decouples orchestration logic from model hosting.

Core Components

• Orchestration Layer (Atlas Reasoning Engine): An event-driven, graph-based cognitive workflow. It decomposes user queries into parallel reasoning tasks (retrieval, planning, action) using a publish-subscribe pattern.
• Prediction Service (Inference Gateway): Acts as a unified entry point. It handles pre/post-processing, implements per-component circuit breakers (to allow graceful degradation if one model fails), and manages request routing.
• Hybrid Backend:
  • Serverless: For lightweight processing or sporadic workloads (scales to zero to eliminate idle costs).
  • Proxy Microservices: For heavier processing, reusing existing monitoring frameworks.
  • Dynamic Autoscaling: GPU instances spin up/down based on real-time demand.

Key Architectural Innovations for Compound Systems

1. Compound-Aware Scaling:
   • Independent Scaling: Models scale based on their specific invocation rates, not aggregate user traffic. This prevents over-provisioning conditional tools while ensuring ubiquitous models (like embeddings) have capacity.
   • Request Priority Queuing: Fast-path models (embeddings) are prioritized over slow-path models (LLMs) to prevent resource starvation.
2. Cascading Cold-Start Mitigation:
   • Coordinated Pre-warming: When a pipeline is accessed, downstream dependencies are warmed in parallel.
   • Tiered Provisioned Concurrency: Critical-path models (LLMs) maintain a baseline of warm instances, while fast models scale from zero.
   • Predictive Warming: Uses traffic signals (e.g., business hours) to pre-load models before demand spikes.
3. Hybrid Deployment Strategy: The system transparently routes traffic between dedicated instances (for high-steady QPS models) and serverless backends (for bursty/conditional models) based on real-time capacity thresholds.
4. MLOps & Observability:
   • Component-Level A/B Testing: Allows swapping individual models (e.g., a new retriever) without redeploying the entire pipeline.
   • Pipeline-Level Metrics: Tracks end-to-end agent latency and cost attribution per pipeline, rather than just per-model metrics.

3. Key Contributions

1. Characterization of Compound-Specific Challenges: The paper identifies and quantifies unique failure modes such as multi-model fan-out overhead, cascading cold-start propagation (multiplicative rather than additive latency), and heterogeneous scaling dynamics.
2. Quantitative Analysis of Parallel Efficiency: Demonstrates that parallel dispatch introduces minimal coordination tax (<2% overhead) compared to sequential invocation.
3. Operational Lessons: Provides a framework for managing enterprise-scale agentic systems, including the necessity of pipeline-aware cost attribution and the superiority of graceful degradation over peak performance.

4. Results & Evaluation

The system was evaluated over 12+ months in production, serving over 8,000 enterprise users and processing ~136 billion tokens in March 2026.

Specific Use Case Findings:

• ApexGuru (Code Analysis): P95 latency halved; costs shifted to actual usage billing.
• Agentforce (Conversational Agents): Linear throughput scaling observed; handled 50+ simultaneous sessions with consistent 5–8s response times.

5. Significance

This paper bridges the gap between theoretical compound AI architectures and practical production deployment. Its significance lies in:

• Shifting the Paradigm: Moving from "scaling models" to "scaling pipelines," recognizing that compound systems require infrastructure that understands dependency graphs and heterogeneous traffic patterns.
• Solving the Cold-Start Chain: Providing a proven solution to the "cascading cold start" problem, which is a primary barrier to the usability of agentic systems.
• Enabling Rapid Iteration: By decoupling components, the architecture allows teams to optimize individual models (A/B testing) in hours rather than weeks, accelerating the development cycle of complex AI agents.
• Cost-Performance Balance: Demonstrating that a hybrid serverless/dedicated approach can drastically reduce costs while maintaining strict SLAs for interactive agents.

The authors conclude that future work should focus on GPU multiplexing, cross-region load balancing, and tighter integration with AI programming frameworks (like DSPy) to further optimize resource allocation across pipeline stages.