Technical Summary: Orcheo – A Modular Full-Stack Platform for Conversational Search

Problem Statement

Conversational Search (CS) research faces two persistent barriers that hinder progress, adoption, and impact:

1. Lack of a Unified Modular Framework: Current advancements in CS components (e.g., query rewriting, dense retrieval, re-ranking) are often developed within isolated, ad hoc pipelines. This fragmentation obscures the true utility of individual contributions, complicates cross-paper comparisons, and creates a "reproducibility gap" where researchers must reinvent underlying infrastructure for every experiment.
2. Barriers to Deployment and Holistic Evaluation: Transitioning from a research prototype to a functional, user-facing application requires engineering skills (backend orchestration, web development) often outside the scope of core IR research. Without standardized deployment tools, researchers struggle to run user studies or apply robust conversational evaluation protocols, creating a bottleneck that prevents theoretical models from being systematically tested.

Methodology: The Orcheo Framework

Orcheo is an open-source Python platform designed to bridge the gap between research and production. It treats CS pipelines as first-class, graph-structured workflow artifacts. The system is built on LangGraph and adheres to three core design principles:

• Minimal Shareable Code: Contributions are packaged as single-file Python modules.
• Composability: New nodes integrate seamlessly with existing upstream/downstream components.
• Type-Safe Interfaces: Strict schema validation (via Pydantic) moves error detection from runtime to development time.

Architecture

Orcheo employs a three-tier node hierarchy:

• BaseNode: Provides core functionality including variable interpolation (e.g., {{node_name.field}}), secure credential resolution, and tool interfaces.
• AINode: Extends BaseNode for LLM interactions (e.g., query rewriting, generation), outputting LangChain message formats.
• TaskNode: Extends BaseNode for utility operations (e.g., retrieval, re-ranking), outputting structured dictionaries merged into the workflow state.

Workflows are defined as directed graphs supporting sequential, conditional, and parallel execution. The system manages state via a typed schema that accumulates results and conversation history.

Full-Stack Platform

Beyond the core framework, Orcheo provides a production-ready infrastructure:

• Dual Execution Modes: Supports local CLI for experimentation and remote backend for production.
• Security & Telemetry: Features an AES-256 encrypted Credential Vault for secure API key management and built-in OpenTelemetry tracing for observability.
• AI-Assisted Development: Integrates with an open-source "Agent Skills" repository to enable "vibe-coding," allowing users to generate workflows and configurations via natural language prompts using AI coding assistants.
• Deployment Tools: Includes Docker Compose configurations, system unit templates, and a ChatKit server integration for embedding full-featured UIs with custom authentication and file handling.

Evaluation Support

Orcheo includes built-in evaluation nodes that can be embedded directly into workflows or used for batch processing. These include:

• ABTestingNode: Routes traffic between pipeline variants.
• LLMJudgeNode: Applies configurable rubrics as real-time quality gates.
• RetrievalEvaluationNode: Computes standard IR metrics (Recall@k, MRR, NDCG, MAP).
• AnswerQualityEvaluationNode: Assesses responses using reference-based metrics and faithfulness checks.

Key Contributions

1. A Modular CS Framework: A LangGraph-based platform allowing researchers to package contributions as plug-and-play, single-file modules, facilitating replication and comparison without monolithic code overhead.
2. Research-to-Production Pipeline: A full-stack environment bridging the prototype-to-system gap via dual execution modes, secure credential management, and automated telemetry.
3. Extensive Starter Kit: A curated library of 45+ off-the-shelf nodes covering the entire CS lifecycle (ingestion, query understanding, retrieval, re-ranking, conversation management, generation, and evaluation), alongside ~100 total built-in nodes.

Results and Validation

The authors validate Orcheo's utility through three case studies:

1. Grounded Generation Pipeline: Demonstrated the composition of a four-node pipeline (QueryRewrite, DenseSearch, ContextCompressor, GroundedGenerator). The entire pipeline required only ~50 lines of Python code, with components configurable via JSON without code changes.
2. QReCC Query Rewriting Evaluation: Implemented a native workflow to evaluate query rewriting on the QReCC benchmark. The system automatically handled conversation threading, metric computation (ROUGE, Semantic Similarity), and reporting.
3. MultiDoc2Dial Grounded Generation Evaluation: Evaluated a full retrieval-generation pipeline on the MultiDoc2Dial benchmark using a nested workflow design where an evaluation node wrapped the generation pipeline as a subgraph.

Engineering Efficiency: Compared to typical manual evaluation scripts requiring 500–1,000+ lines of code, equivalent Orcheo workflows required only 85–150 lines. This reduction is attributed to composable metric nodes, automatic history management, and configuration-driven execution.

Performance: The paper reports corpus-level scores for QReCC and MultiDoc2Dial using Orcheo workflows with GPT-4o-mini. While the absolute scores (e.g., Token F1 of 8.34 on MultiDoc2Dial) are lower than published reference results (48.04), the authors note these references use different models (SARCAT) and settings. The primary validation is the successful execution of complex, multi-stage pipelines with minimal engineering overhead.

Significance and Claims

The paper claims Orcheo's primary value lies in shifting the community from "code sharing" to "system sharing." By unifying algorithms with execution environments, Orcheo ensures that experiments remain executable and extensible after publication.

The authors position Orcheo as a solution to the "reproducibility gap" and the engineering burden of deployment. It is designed not just as a toolkit, but as a shared infrastructure that:

• Lowers the barrier to entry for CS research.
• Provides a stable foundation for establishing reliable baselines.
• Enables rapid bootstrapping of complete CS pipelines.
• Facilitates knowledge transfer by allowing researchers to share single-node contributions that integrate immediately into existing workflows.

The paper acknowledges deliberate trade-offs, noting that performance on massive-scale document collections has not been extensively benchmarked and that the modular architecture requires researchers to make integration decisions that monolithic systems handle implicitly. However, the authors argue these are necessary costs for the transparency and extensibility required by the research community.