AnovaX: A Local, Multi-Agent Voice Assistant with LLM Planning, Typed Executors, and Adaptive Recovery

Technical Summary: AnovaX – A Local, Multi-Agent Voice Assistant

Problem Statement
Current desktop voice assistants (e.g., Siri, Alexa, Google Assistant) rely on cloud pipelines that transmit raw audio off-device and offer a fixed, closed set of skills. While large language models (LLMs) have solved the problem of understanding natural language, existing systems fail to provide users with visibility, control, or the ability to act directly on the local desktop environment. Users cannot inspect the reasoning behind a command, add custom skills easily, or ensure that personal audio and data remain local. There is a gap for a system that is local-first, capable of interacting with real desktop applications (opening, typing, hotkeys), and fully inspectable by a human operator.

Methodology
AnovaX is a local-first voice assistant implemented in approximately 1,800 lines of Python and one HTML file. It runs entirely on the user's machine, treating the desktop as its action surface. The system architecture consists of the following core components:

1. Input and Wake-Word Gating: A background thread uses the SpeechRecognition library to listen for a specific wake phrase ("anova" + trigger word). Only recognized phrases are routed to the planning engine; ambient chatter is discarded.
2. LLM Planner (Static Path): The core logic uses an LLM (Gemini) to generate a single JSON plan containing a sequence of tool calls, a final spoken response, and optional memory updates. The system prompt enforces a strict JSON schema.
3. Two-Stage Safety Filter: Before execution, every plan (including recursive sub-plans) undergoes:
   • Prompt-level constraints: Explicit instructions in the system prompt to avoid dangerous actions.
   • Python-level enforcement: A safety_check() function that validates against a whitelist of allowed tools, a denylist of dangerous keywords (e.g., rm -rf, sudo), and bounds on plan size (max 8 steps, 2 levels of recursion).
4. Multi-Agent Orchestrator: Instead of a linear executor, the plan is dispatched to a Python orchestrator that spawns typed child agents.
   • Typed Agents: Ten specific agent classes (e.g., AppAgent, TypingAgent, BrowserAgent) correspond to tools. Each has specific Time-To-Live (TTL) limits, retry policies, and resource locks.
   • Concurrency: Agents are batched and executed on a bounded thread pool (capped at 8 workers). Read-only tools (e.g., get_time, screenshot) are parallelized; input tools (typing, clicking) are serialized via kb_mouse and tts locks to prevent race conditions.
   • Recursive Planning: A MetaAgent allows the LLM to delegate sub-goals back to itself, capped at two levels of nesting.
5. Adaptive Recovery Loop: If a "core" step fails (e.g., an app fails to open), the system switches from the static path to an autonomous recovery loop.
   • This loop uses a compact ReAct-style prompt to generate a new batch of actions.
   • It includes speculative parallelism: while the LLM decides the next step, the system pre-executes read-only tools from the original failed plan in the background, hiding latency.
   • The loop is bounded (max 6 turns, 20 total agents) and re-runs the safety filter on every iteration.
6. Memory and Observability:
   • Three-layer memory: RAM-based conversation history, session state (recently opened apps), and a persistent JSONL fact store.
   • Mobile Companion: A Flask server exposes the assistant over local WiFi. It streams agent lifecycle events (via Server-Sent Events) and the laptop screen (via MJPEG) to a phone, allowing remote voice control and visual verification of actions.

Key Contributions
The paper presents AnovaX as a system description rather than a benchmark study, highlighting four primary contributions:

• Local Multi-Agent Orchestrator: A system where an LLM plans once, and a Python parent process dispatches typed, bounded child agents with specific retry policies and locks, rather than letting the LLM directly control the OS.
• Two-Stage Safety Filter: A robust defense mechanism combining prompt instructions with a strict Python-based whitelist/denylist that applies to both static and recursively generated plans.
• Adaptive Recovery with Speculative Parallelism: A recovery mechanism that handles core failures via a bounded ReAct loop while simultaneously pre-running read-only tools to mask LLM latency.
• Full Observability and Remote Control: A design that mirrors every agent state transition to a mobile device and streams the desktop screen, allowing users to verify actions remotely without the LLM ever directly accessing the keyboard.

Results and Evaluation
The evaluation is qualitative, based on a single Windows 11 laptop with 16GB RAM.

• Capability: The system successfully handles diverse tasks including launching apps, typing into focused windows, web searches, taking screenshots, and executing multi-step compound commands (e.g., "open calc, type 21+21, press enter"). It correctly refuses dangerous commands (e.g., deleting folders).
• Latency: End-to-end latency is dominated by network-dependent components (Google Speech Recognition: 0.6–1.4s; Gemini planning: 0.7–2.0s). The local orchestrator adds negligible overhead (<0.05s).
• Ablation Studies:
  • Disabling the LLM (regex fallback) reduces capability to simple commands but maintains safety.
  • Disabling memory breaks context resolution (e.g., "type in it").
  • Disabling locks causes race conditions in multi-step typing tasks.
  • Disabling the recovery loop results in permanent failure on minor errors (e.g., wrong app name), whereas the loop successfully adapts.

Significance and Claims
The authors claim that AnovaX demonstrates that a "legible, few-thousand-line assistant" is sufficient to open applications, type into them, coordinate concurrent actions, and recover from failures without relying on cloud pipelines for action execution. The significance lies not in novel algorithms, but in the ratio and architecture: combining a single LLM planning call with a bounded, typed, multi-agent runtime and a strict safety filter.

The paper explicitly states that the goal is not to compete with commercial assistants in scale or accuracy, but to show that a system can be built where:

1. Audio and personal facts stay local (unless the user opts in).
2. The assistant acts on real desktop applications.
3. Every tool call is inspectable, whitelisted, and rejectable.
4. The system is driven entirely from a remote device (phone) while the user watches the execution in real-time.

The authors acknowledge limitations, including the "blind" nature of the executor (no screen perception), reliance on cloud APIs for speech and planning (though local alternatives are noted as future work), and the inherent risks of the MJPEG stream and PIN-based authentication. They emphasize that the current design prioritizes legibility and auditability over squeezing out marginal accuracy gains.