VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agents

Technical Summary: VIDEE – Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agents

Problem Statement
While Large Language Models (LLMs) and agentic systems have lowered technical barriers for text analytics, fully autonomous agents currently face significant reliability issues when handling end-to-end text analysis tasks. Preliminary experiments with coding agents (e.g., Claude Code) revealed critical failure modes, including:

• Over-simplification: Agents often present ill-considered plans as definitive, ignoring the large decomposition space and trade-offs between subtask difficulty and pipeline robustness.
• Hallucination and Unreliability: Agents frequently hallucinate execution results, fabricate data, or fail to execute generated code, particularly in complex pipeline scenarios.
• Lack of Observability: Conversational interfaces obscure intermediate steps, making it difficult for humans to audit reasoning, detect silent data loss, or verify analytical decisions.
• Technical Knowledge Gaps: Analysts with varying levels of expertise struggle to keep pace with rapid advancements (e.g., RAG, generative reasoning), leading to under-utilization of state-of-the-art methods.

Consequently, there is a need for a system that enables human oversight and intervention without requiring deep NLP expertise, bridging the gap between autonomous automation and human insight.

Methodology
The authors propose VIDEE, a system that operationalizes a three-stage human-agent collaboration workflow: Decomposition, Execution, and Evaluation.

1. Decomposition (Planning):

   • Workflow: A human user defines a goal and dataset. A "decomposer agent" searches a large space of possible plans using a Monte-Carlo Tree Search (MCTS) algorithm adapted for generative reasoning.
   • Human-in-the-Loop: The search process is visualized as a tree where nodes represent "semantic tasks" (high-level conceptual descriptions). Users can intervene by adjusting scoring criteria (Complexity, Coherence, Importance), selecting expansion nodes, or redirecting the search.
   • Reward Mechanism: A committee of LLM judges (GPT-4o, Claude-3.5-Sonnet, Gemini-2.0) evaluates nodes based on the three criteria. User feedback on scores is stored as few-shot prompts to refine future evaluations.

2. Execution (Implementation):

   • Workflow: Once a plan is selected, an "executor agent" converts semantic tasks into "primitive tasks" (concrete implementations with specific algorithms and parameters).
   • Architecture: The system uses a JSON-based specification language and a MapReduce conceptual framework to manage data structure changes (Analysis Units). This ensures that outputs from one task (e.g., extracted entities) are correctly formatted as inputs for subsequent tasks.
   • Interface: Users inspect the generated pipeline, modify parameters (e.g., prompt templates, hyperparameters), and execute steps sequentially. The system handles code generation and compilation automatically.

3. Evaluation (Validation):

   • Workflow: An "evaluator agent" runs LLM-based judges to assess execution results against user-defined or recommended criteria.
   • Visualization: Results are visualized using bar charts for categorical distributions and an extended Topic Radial Chart. The radial chart maps documents to topic regions and further subdivides them by evaluation categories (e.g., "Long" vs. "Short" summaries), maintaining visual continuity from the initial sensemaking phase.

Key Contributions

1. A Three-Stage Human-Agent Collaboration Workflow: A structured paradigm that separates decomposition, execution, and evaluation, explicitly defining the division of labor between human oversight and agent automation.
2. VIDEE System: An agentic interface featuring:
   • A Searching View with MCTS-based tree visualization for planning.
   • A Dataset Inspection View with a Topic Radial Chart for sensemaking.
   • Execution and Evaluation Views that support no-code pipeline construction, parameter tuning, and LLM-judge-based validation.
3. Empirical Findings:
   • Quantitative Evaluation: Generated pipelines were found to be comparable to human-crafted pipelines (LLooM, TnT-LLM) in an "Arena" style evaluation. In a specific task on the Wikipedia dataset, the system achieved an 83% concept coverage rate, significantly outperforming BERTopic (52.6%) and GPT-4o alone (53%).
   • User Study: A study with six participants (ranging from novices to experts) demonstrated that the system is learnable within 30 minutes. Users reported that the visual workflow reduced cognitive load, helped in error detection via explicit "Analysis Units," and fostered trust through procedural transparency compared to black-box GenAI systems.

Results and Significance
The paper claims that VIDEE successfully addresses the unreliability of fully autonomous agents by introducing human oversight at critical junctures. The system demonstrates that:

• Democratization: It lowers the barrier for entry-level analysts to perform complex text analytics, though it still requires a learning curve suitable for professionals (e.g., journalists, social scientists) rather than the general public.
• Trust and Alignment: By treating LLM judges as "consultants" rather than ground truths, and allowing users to manage the process, the system fosters trust without requiring perfect alignment between human and AI judgments.
• Visual Scaffolding: Visualizing the decomposition space and data flow (via trees and radial charts) is essential for making agent actions predictable and controllable.

The authors conclude that while current coding agents are not yet ready for fully autonomous end-to-end text analytics, a human-agent collaborative approach with robust visualization can significantly improve reliability, reproducibility, and the quality of insights derived from textual data. The work suggests future directions such as integrating conversational agents for pipeline modification and extending evaluation methods to non-textual tasks like clustering.