💬 NLP

From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems

This survey defines Compound AI Systems as an emerging paradigm that integrates large language models with external components like retrievers and agents, proposing a unified taxonomy and analysis of foundational paradigms such as RAG and LLM agents to address current fragmentation and guide future research in system-level artificial intelligence.

Original authors: Jiayi Chen, Junyi Ye, Guiling Wang

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiayi Chen, Junyi Ye, Guiling Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, well-read librarian named "LLM" (Large Language Model). This librarian has read almost every book ever written up to a certain date. They are incredibly fast at writing stories, answering questions, and chatting. However, they have three big problems:

They forget things: They can't remember facts that happened after they stopped studying (their knowledge is "stale").
They make things up: Sometimes, when they don't know the answer, they confidently invent a fake fact (this is called "hallucination").
They can't do math or use tools: They are great at talking, but they can't actually calculate a complex equation, search the live internet, or control a robot arm.

This paper introduces a new way of building AI called Compound AI Systems (CAIS). Instead of relying on just the one librarian, CAIS builds a team or an orchestra around the librarian to fix these problems.

Here is how the paper breaks down this "team," using simple analogies:

1. The Four Key Team Members (The Axes)

The paper organizes this new AI world into four main roles that work together:

The Researcher (RAG - Retrieval-Augmented Generation):
- The Problem: The librarian doesn't know today's news or your company's private files.
- The Solution: Before the librarian answers, a Researcher runs to the library shelves (or the internet) to find the specific, up-to-date documents. They hand these notes to the librarian so they can answer based on facts, not guesses.
- Analogy: It's like giving the librarian a cheat sheet right before a test.
The Project Manager (LLM Agents):
- The Problem: The librarian is good at one task but gets confused by a long, complicated project that needs many steps.
- The Solution: An Agent is a version of the librarian that can plan. It breaks a big task (like "Plan a vacation") into small steps (Book flight -> Find hotel -> Check weather). It can also use tools, like calling a travel API or doing math, and then check its own work to fix mistakes.
- Analogy: It's like a foreman who doesn't just lay bricks but draws the blueprints, orders the materials, and inspects the work.
The Multi-Sensory Expert (MLLMs - Multimodal LLMs):
- The Problem: The librarian can only read text. If you show them a picture of a broken engine or a video of a storm, they are blind.
- The Solution: These are librarians who have been given eyes and ears. They can look at images, listen to audio, and watch videos, then describe what they see or answer questions about it.
- Analogy: It's like giving the librarian glasses and headphones so they can understand a world that isn't just written words.
The Conductor (Orchestration):
- The Problem: If you have a Researcher, a Project Manager, and a Multi-Sensory Expert, they might all talk over each other or get confused about who does what.
- The Solution: The Conductor is the system that manages the whole team. It decides when to call the Researcher, when to ask the Agent to plan, and how to combine their answers into one final result. It makes sure the team works smoothly together.
- Analogy: Think of a symphony orchestra. The librarian is the violinist, but the Conductor makes sure the drums, flutes, and violins all play at the right time to make music, not noise.

2. How They Work Together (The Pipeline)

The paper explains that a real-world Compound AI System isn't just one of these things; it's usually all of them at once.

Imagine you ask the system: "Analyze this photo of a broken machine and find the manual to fix it."

The Multi-Sensory Expert looks at the photo and understands what the machine is.
The Researcher goes out and finds the specific manual for that machine model.
The Project Manager (Agent) figures out the steps to fix it and decides to use a tool to check if the part is in stock.
The Conductor (Orchestration) manages the flow, making sure the photo analysis happens before the manual search, and combines everything into a clear answer for you.

3. The Current Challenges

The paper admits that while this "team" approach is powerful, it's also messy right now:

It's complicated: Building a team is harder than hiring one person. If one part breaks, the whole system might fail.
They don't speak the same language: Different tools and agents often use different formats, making it hard to connect them (like trying to plug a European charger into an American socket).
It's hard to test: We don't have perfect tests yet to see if the whole team is working well, especially when they are looking at pictures and talking to tools at the same time.

4. The Future: Standardization

The paper highlights that people are starting to create "universal adapters" (like the Model Context Protocol or MCP). These are like universal power strips that let any AI tool plug into any AI system easily. The goal is to stop building custom bridges for every single connection and instead have a standard way for all these AI components to talk to each other.

Summary

In short, this paper says: Stop trying to make one giant brain that does everything perfectly. Instead, build a smart system that connects a brain (the LLM) with a memory bank (Retrieval), a planner (Agents), eyes/ears (Multimodal), and a manager (Orchestration). This "Compound" system is the future of AI because it overcomes the limits of any single model.

Technical Summary: From Standalone LLMs to Integrated Intelligence: A Survey of Compound AI Systems

1. Problem Statement

While Large Language Models (LLMs) have evolved into foundational AI infrastructure, they possess inherent structural limitations that hinder their deployment in dynamic, real-world environments. These limitations include hallucination (fluent but factually inaccurate output), staleness (inability to access post-training knowledge), and bounded reasoning (finite context windows constraining multi-hop reasoning and long-horizon task decomposition).

To overcome these barriers, the community is converging on Compound AI Systems (CAIS). CAIS are defined as modular, extensible architectures that integrate LLMs with specialized external components—such as retrievers, tool-using agents, symbolic planners, memory modules, and multimodal encoders. However, despite growing adoption, the CAIS landscape remains fragmented. Existing literature addresses isolated components (e.g., specific RAG surveys, agent frameworks, or optimization methods) but lacks a unified, system-level framework for analysis, taxonomy, and evaluation that synthesizes the interactions across the entire stack.

2. Methodology

The authors conducted a systematic survey based on a structured literature search across four major scholarly databases (ACM Digital Library, arXiv, IEEE Xplore, Semantic Scholar) and manual inspection of top-tier conference proceedings (ACL, NeurIPS, ICML, CVPR, etc.).

Search Scope: Publications from 2020 to early 2026, targeting the period following the introduction of GPT-3 and seminal RAG work.
Selection Criteria: Papers were included if they used an LLM as a primary reasoning component combined with at least one external module (retriever, tool, memory, API, encoder, or orchestration framework) and presented novel empirical results, architectures, or systematic analyses.
Final Corpus: Approximately 220 papers were retained and analyzed.
Taxonomy Construction: The authors organized the landscape along four orthogonal axes: Retrieval-Augmented Generation (RAG), LLM Agents, Multimodal LLMs (MLLMs), and Orchestration.

3. Key Contributions

A. Multi-Dimensional Taxonomy and Unified Framework

The paper proposes a formal representation of CAIS as a function $f(L, C, D)$ , where:

$L$ : The set of LLMs (general or fine-tuned).
$C$ : The set of external components (tools, retrievers, encoders, etc.).
$D$ : The system design defining architecture, topology, and coordination.

The survey integrates four foundational axes into a cohesive framework:

RAG: Grounding LLMs in external non-parametric memory.
LLM Agents: Autonomous decision-making, planning, and tool use in iterative loops.
MLLMs: Extending LLMs to perceive and reason over non-textual modalities (images, audio, video).
Orchestration: The structural and coordination design integrating all components.

B. Comparative Analysis and Trade-offs

The paper provides four structured comparison tables distilling design trade-offs for each axis:

RAG Retriever Paradigms (Table 2): Compares Sparse, Dense, Graph, Hybrid, and LLM-as-Retriever approaches across latency, recall, scalability, interpretability, and domain specificity. It highlights that Hybrid retrievers often achieve the highest recall, while LLM-as-retriever offers flexibility at high inference costs.
LLM Agent Reasoning Frameworks (Table 3): Analyzes frameworks like Chain-of-Thought, ReAct, Tree of Thoughts, Reflexion, and Self-RAG. It notes that while ReAct is the most widely adopted baseline for tool use, Tree of Thoughts incurs high compute costs for deep planning, and Reflexion offers strong error recovery via memory.
MLLM Architectures (Table 5): Compares fusion strategies (Cross-attention, Q-Former bridging, Linear projection, Native end-to-end). It identifies that while native end-to-end training (e.g., Gemini) offers strong reasoning, modular approaches (e.g., LLaVA, BLIP-2) offer better efficiency and upgradeability.
Orchestration Frameworks (Table 7): Evaluates frameworks like LangChain, AutoGen, DSPy, and CrewAI based on topology, coordination mechanisms, and tool support.

C. Standardization and Interoperability Landscape

The survey maps the emerging standardization stack as a three-layer architecture:

Schema Layer: Structured tool interfaces (e.g., OpenAI function calling, JSON Schema).
Tool-Access Protocol Layer: The Model Context Protocol (MCP), which standardizes communication between AI systems and external tool servers.
Agent Interoperability Layer: Cross-framework protocols including Agent-to-Agent (A2A), Agent Communication Protocol (ACP), Agent Network Protocol (ANP), and Agent Spec.
The authors argue that interoperability is converging through "translation and layering" (adapters and intermediate representations) rather than a single universal standard.

D. Evaluation Methodologies

The paper reviews benchmarks across the four axes, noting a persistent gap in unified system-level metrics.

RAG: Evaluated on retrieval quality (MRR, nDCG) and generation factuality (EM, F1) using datasets like HotpotQA and BEIR.
Agents: Assessed on role fidelity, reasoning trace accuracy, and tool-use efficiency.
MLLMs: Tested on visual QA, document understanding, and safety alignment (e.g., MM-SafetyBench).
Orchestration: Measured via infrastructure performance (throughput, latency) and system coordination metrics.

4. Results and Findings

System Complexity: CAIS introduce significant engineering overhead. Debugging and optimization become super-linearly difficult as the number of integrated components increases.
Failure Modes: The survey identifies specific failure modes in MLLMs (visual hallucination, cross-modal inconsistency) that propagate errors through the entire CAIS pipeline.
Design Trade-offs: There is no single optimal configuration. For instance, while dense retrievers capture semantic similarity, they lack the interpretability of sparse methods; similarly, end-to-end MLLM training yields strong performance but sacrifices modularity.
Standardization Gaps: While protocols like MCP are gaining traction, security governance (authentication, sandboxing) for standardized interfaces remains under-developed, introducing new attack surfaces.

5. Significance and Claims

The paper positions itself as the first comprehensive, cross-axis systems-level synthesis of Compound AI Systems. Its significance lies in:

Unifying Fragmented Research: It bridges the gap between isolated surveys on RAG, agents, and multimodal models, providing a shared vocabulary and conceptual framework.
Providing a Design Blueprint: By formalizing the $f(L, C, D)$ model and analyzing trade-offs, it offers researchers and practitioners a foundation for designing next-generation AI systems that move beyond "standalone soloists" to "orchestrators of heterogeneous AI ensembles."
Identifying Critical Challenges: It explicitly outlines open research directions, including the need for end-to-end trainable pipelines, unified evaluation metrics for interactive/multimodal tasks, and robust governance frameworks for interoperability.

The authors conclude that as CAIS expand into real-world, multimodal, and interactive environments, a systems-level perspective synthesizing retrieval, agency, perception, and orchestration is essential for advancing the field.