🤖 AI

INFRAMIND: Infrastructure-Aware Multi-Agent Orchestration

INFRAMIND is a reinforcement learning-based framework that optimizes multi-agent LLM orchestration by dynamically adapting topology planning, model routing, and request scheduling to real-time infrastructure conditions, thereby significantly improving accuracy, reducing latency, and maintaining strict service-level objectives under high concurrent loads.

Original authors: Ahasan Kabir, Jiaqi Xue, Mengxin Zheng, Qian Lou

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ahasan Kabir, Jiaqi Xue, Mengxin Zheng, Qian Lou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a busy restaurant kitchen. You have a team of chefs (the AI models) with different skill levels: some are master chefs who can cook complex dishes perfectly but take a long time, while others are quick line cooks who can handle simple orders fast.

In the current way most AI systems work (what the paper calls "Infrastructure Blindness"), the manager assigns orders based only on what the customer wants, ignoring the current state of the kitchen.

The Problem: The "Blind" Manager

If a customer orders a complex steak, the manager blindly sends it to the Master Chef because that chef is the best at steak. But the manager doesn't look to see that the Master Chef is already buried under 100 other orders, creating a massive line. Meanwhile, a very capable Line Cook who could have handled the steak just fine is standing around with nothing to do.

This causes two big issues:

The Bottleneck: The Master Chef's line gets so long that the customer waits 30 minutes for a steak that could have been ready in 5 minutes.
Wasted Talent: The Line Cook sits idle, even though they could have helped, because the manager never checks if they are free.

In the world of AI, this happens when multiple AI agents work together. If one AI gets stuck in a "queue" (waiting to be processed), the whole chain of reasoning slows down, and the system wastes expensive computing power.

The Solution: INFRAMIND (The "Smart" Manager)

The paper introduces INFRAMIND, a new system that acts like a super-smart kitchen manager who constantly watches the kitchen's live status. It makes three key decisions differently:

1. The Planner (The Menu Designer)

Old Way: Decides the recipe structure based only on the order. "This is a complex dish, so we need a 5-step plan with the Master Chef."
INFRAMIND: Looks at the kitchen first. "The Master Chef is swamped. Let's simplify the plan. We'll use a shorter, simpler recipe that the Line Cook can handle quickly."
Analogy: If the kitchen is chaotic, the manager switches from a 10-course tasting menu to a quick, delicious sandwich. If the kitchen is quiet, they go all out with the fancy 10-course meal.

2. The Executor (The Order Runner)

Old Way: Sends every request to the "best" model, regardless of how busy they are.
INFRAMIND: Checks the line at every single step. "The Master Chef's line is full, but the Line Cook's line is empty. Let's send this step to the Line Cook. Also, since we saved time by skipping the long line, let's ask the Line Cook to think a little deeper (use 'DeepThink') to make sure the answer is still high quality."
Analogy: Instead of forcing a customer to wait in a long line for a VIP, the manager directs them to a fast lane where a capable staff member is waiting, ensuring they get a great result without the wait.

3. The Scheduler (The Priority List)

Old Way: First-Come, First-Served. If a customer with a 5-minute deadline arrives after a customer with a 1-hour deadline, the slow one gets served first.
INFRAMIND: Uses an "Earliest Deadline First" rule. It looks at the customer's time limit. "This customer needs food in 2 minutes! Move them to the front of the line, even if they arrived later."
Analogy: It's like an emergency room triage. The person with the most urgent need gets treated first, not the person who arrived first.

The Results: Why It Matters

The paper tested this system on five different difficult tasks (like math, coding, and logic puzzles) under different levels of traffic (from a quiet kitchen to a rush hour).

At Low Traffic: INFRAMIND was more accurate than the old systems because it knew it had extra time and could use the "Master Chefs" for deep thinking. It was also up to 7 times faster.
At High Traffic (The Rush Hour): This is where the old systems failed. They kept sending orders to the same busy chefs, causing lines to get so long that the system crashed (99% of requests failed to finish on time). INFRAMIND, however, kept the kitchen running smoothly. It maintained 99.9% success rate even when the other systems dropped below 50%.

The Bottom Line

INFRAMIND fixes the "blindness" of current AI systems. It doesn't just ask, "Which AI is smartest?" It asks, "Which AI is smartest AND available right now?" By watching the real-time traffic and queues, it balances speed and quality automatically, ensuring that AI systems don't get stuck in traffic jams even when they are under heavy load.

Technical Summary: INFRAMIND

1. Problem Statement: Infrastructure Blindness

Current multi-agent Large Language Model (LLM) orchestration systems suffer from a critical failure mode termed infrastructure blindness. While existing methods (e.g., MoA, GPTSwarm, MasRouter) have advanced in learning task-adaptive topologies and model selection based on static query features, they operate without observing the runtime state of the serving infrastructure.

On shared GPU clusters serving multiple models (e.g., via vLLM or SGLang), this blindness leads to systematic resource underutilization and latency explosions:

Queue Imbalance: "Preferred" models accumulate deep request queues while equally capable alternatives sit idle.
Compounding Delays: In multi-agent pipelines, a bottleneck at any single step delays every downstream agent, causing latency to compound.
Static Decisions: Routing decisions are made based on training-time information regimes that are disconnected from runtime conditions (queue depths, KV-cache pressure, and latencies).
Inflexible Reasoning: Systems fail to adapt reasoning depth (e.g., switching from "DeepThink" to "Flash") based on available capacity, leaving idle resources untapped or overloading congested ones.

The core challenge is that infrastructure signals are dynamic and noisy, yet they must drive three coupled decisions: planning (topology/roles), per-step routing (model/strategy selection), and scheduling (queue prioritization).

2. Methodology: INFRAMIND Framework

The authors propose INFRAMIND, the first end-to-end infrastructure-aware multi-agent orchestrator. It integrates real-time system metrics into every layer of the orchestration stack, formulated as a Hierarchical Constrained Markov Decision Process (CMDP) and solved via Reinforcement Learning (RL).

Core Components

Infra-Aware Planner:
- Function: Determines the reasoning structure (topology, agent count, roles) at query arrival ( $t=0$ ).
- Mechanism: Unlike prior work that relies solely on query embeddings, the planner conditions its decisions on a summary of current system load ( $z_0$ ) and remaining budget.
- Implementation: Uses a cascaded controller where Feature-wise Linear Modulation (FiLM) injects system state vectors (budget, queue depth, E2E latency, KV cache) into the query embedding. This biases the planner toward simpler graphs under congestion and richer topologies when capacity is available.
Infra-Aware Executor:
- Function: Selects the specific model and reasoning strategy (Flash, Concise, DeepThink) for each agent step.
- Mechanism: Operates on a dual-pathway architecture:
  - Semantic Pathway: Processes query and role embeddings (what the task needs).
  - Resource Pathway: Processes remaining budget and live system metrics (what the system can deliver).
- Decision: Merges these signals to jointly select a model and strategy. For example, if a large model's queue is saturated, the executor may redirect to an idle smaller model and compensate by using a "DeepThink" strategy, leveraging the time saved by skipping the queue to maintain accuracy.
Budget-Aware Priority Scheduler:
- Function: Manages request ordering within each model's local queue.
- Mechanism: Implements an Earliest-Deadline-First (EDF) policy. Each query is assigned a deadline based on its arrival time and time budget. Every agent call inherits this deadline.
- Impact: Prevents tight-budget requests from being blocked behind relaxed ones (head-of-line blocking), ensuring urgent queries are served first.

Training Approach

The three components are trained jointly as a single hierarchical RL policy under a shared budget constraint.

Objective: Maximize quality (solved rate) subject to a latency budget.
Optimization: Uses a Lagrange multiplier ( $\lambda$ ) to automatically discover the quality-latency trade-off. The executor uses PPO (Proximal Policy Optimization) for per-step trajectories, while the planner uses REINFORCE on episode returns.
Adaptation: The system is trained across varying budget tiers and Poisson arrival rates, allowing it to learn to adapt to different congestion regimes.

3. Key Contributions

Identification of Infrastructure Blindness: The paper empirically quantifies how ignoring runtime infrastructure state leads to queue imbalances, avoidable latency, and resource underutilization in multi-agent systems.
End-to-End Infrastructure-Aware Orchestration: Introduction of INFRAMIND, which unifies planning, execution, and scheduling into a single RL policy that conditions on real-time telemetry (queue depth, KV cache, latency).
Performance Gains: Demonstrated significant improvements in accuracy, latency, and Service Level Objective (SLO) compliance across diverse benchmarks and load conditions.

4. Experimental Results

The system was evaluated on five benchmarks (MBPP, HumanEval, GSM-Hard, MATH, MMLU-Pro) using a heterogeneous pool of five models (3B to 32B parameters) served on shared GPUs.

Low Load ( $\mu=10$ req/min): INFRAMIND achieved up to +7.6 percentage points (pp) higher accuracy than the strongest baseline (MasRouter) while reducing latency by up to 7×. For instance, on MATH, it reached 82.0% accuracy vs. 74.4% for MoA.
High Load ( $\mu=100$ req/min): This is where the system's value is most pronounced.
- SLO Compliance: INFRAMIND sustained up to 99.9% SLO compliance (queries completing within the 300s budget) on HumanEval.
- Baseline Collapse: All baselines dropped below 50% SLO compliance under high load. MoA and GPTSwarm fell below 12% on most benchmarks, with latencies exceeding 1,000 seconds.
- Latency: INFRAMIND maintained latencies under 300s across all benchmarks, whereas baselines migrated into "severely congested" zones.
Blackbox/Hybrid Pools: The approach was extended to API-only and hybrid pools (mixing whitebox and API models) using client-side proxies for congestion signals. INFRAMIND maintained robust performance where baselines collapsed, proving that infrastructure awareness does not require privileged server access, only a signal correlated with responsiveness.

5. Significance and Claims

The paper claims that INFRAMIND addresses a fundamental gap in multi-agent LLM systems: the disconnect between task-level intelligence and infrastructure reality. By making infrastructure awareness a first-class citizen at the planning, execution, and scheduling layers, the system automatically balances quality against latency without hand-coded rules.

The authors assert that this approach is critical for production multi-agent workflows running on shared GPU clusters (e.g., enterprise internal suites or web-scale deployments), where queue contention and cache pressure are the primary drivers of tail latency. The framework is presented as a general solution that can be deployed across different serving backends (vLLM, SGLang, or API providers) to significantly improve SLO compliance and resource utilization.