💻 computer science

OpenJarvis: Personal AI, On Personal Devices

OpenJarvis introduces a decomposed, typed personal AI architecture that leverages LLM-guided spec search to optimize five independent primitives, enabling on-device systems to match cloud-level accuracy while drastically reducing API costs and latency.

Original authors: Jon Saad-Falcon, Avanika Narayan, Robby Manihani, Tanvir Bhathal, Herumb Shandilya, Hakki Orhun Akengin, Gabriel Bo, Andrew Park, Matthew Hart, Caia Costello, Chuan Li, Christopher Ré, Azalia Mirhosei

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jon Saad-Falcon, Avanika Narayan, Robby Manihani, Tanvir Bhathal, Herumb Shandilya, Hakki Orhun Akengin, Gabriel Bo, Andrew Park, Matthew Hart, Caia Costello, Chuan Li, Christopher Ré, Azalia Mirhoseini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, world-class personal assistant who lives in the cloud. They are incredibly smart, can solve complex problems, and handle your sensitive data (like your emails, calendar, and health records) by sending them to a massive server farm. This works well, but it costs a lot of money every month, requires an internet connection, and means you are handing your private diary to a stranger to read.

Now, imagine you want that same assistant to live inside your own laptop or phone. You want it to be private, free of monthly fees, and work even when you're on a plane with no Wi-Fi.

The problem is that if you just take the "brain" of the cloud assistant (a powerful AI model) and try to run it on your phone, it crashes. It's like trying to drive a Formula 1 race car engine in a compact sedan; the car wasn't built for that engine. The car's suspension, fuel system, and steering (the software stack) are all tuned for the smaller engine. If you swap the engine without changing the rest of the car, it falls apart.

This is the core problem the paper "OPENJARVIS" solves.

The Problem: The "Engine Swap" Failure

The researchers tried a simple experiment: They took existing personal AI systems (like OpenClaw or Hermes Agent) and simply swapped the cloud brain for a local one (a smaller AI model that runs on your device).

The result? The system became terrible. Accuracy dropped by 25–39%.
Why? Because the "instructions," "tools," and "memory settings" were all written specifically for the giant cloud brain. When you put a local brain in, it gets confused by instructions it wasn't designed to follow.

The Solution: The "LEGO Blueprint" (The Spec)

The authors realized you can't just swap the brain; you have to rebuild the whole car to fit the new engine. To do this, they created OPENJARVIS.

Think of OPENJARVIS as a universal LEGO blueprint (called a "Spec"). Instead of a messy, tangled knot of code, they broke the personal AI system into five distinct, interchangeable LEGO blocks:

Intelligence: The brain (the AI model).
Engine: The car's engine and transmission (how the model runs).
Agents: The driver's logic (how it thinks and decides).
Tools & Memory: The map and toolbox (how it accesses files and the internet).
Learning: The mechanic (how the system improves itself).

Because these are separate blocks, you can swap the "Intelligence" block for a local model and then tune the other four blocks to match it perfectly. It's like taking a race car engine and adjusting the suspension, tires, and fuel mix specifically for that engine, rather than trying to force it into a sedan.

The Magic Trick: The "Tutor and Student" Search

Even with the LEGO blocks, a local brain is still less smart than a cloud brain. How do you make the local one perform as well as the cloud one?

They invented a method called LLM-guided spec search. Imagine a Tutor-Student relationship:

The Student: The local AI running on your device.
The Tutor: A super-smart cloud AI (like the ones you pay for).

Here is how they work together:

The Test: The Student tries to do a task and fails.
The Diagnosis: The Tutor looks at the failure and says, "Ah, you failed because you didn't use the calendar tool correctly," or "You need to change your reasoning steps."
The Edit: The Tutor doesn't just give the answer; it rewrites the LEGO blueprint. It might say, "Change the tool description," "Adjust the memory settings," or "Tweak the brain's parameters."
The Gatekeeper: A strict rule checks: "Did this change make the Student better without making it worse at other things?" If yes, the change is accepted. If no, it's rejected.
The Result: The Student learns and improves its blueprint.

Crucially: The Tutor only helps during the "search" phase (the setup). Once the blueprint is perfected, the Student runs entirely on your device with zero cloud calls. The cloud Tutor disappears, and you are left with a highly optimized, private, local assistant.

The Results: Private, Fast, and Cheap

The paper tested this on 8 different real-world tasks (coding, research, scheduling, etc.). Here is what they found:

Performance: The optimized local assistants performed almost as well as the expensive cloud ones. On 4 out of 8 tests, the local version was actually better or equal. On average, they were only 3.2% less accurate than the best cloud model.
Cost: Because the local model runs on your own hardware, the cost per query dropped by roughly 800 times. You stop paying the monthly subscription.
Speed: The local version was 4 times faster because it didn't have to wait for data to travel to the cloud and back.
Privacy: Your data never leaves your device during normal use.

Summary

OPENJARVIS is a new way to build personal AI. Instead of forcing a local AI to work like a cloud AI, it breaks the system into parts and uses a "Tutor" to redesign those parts specifically for the local hardware. The result is a personal AI that lives in your pocket, keeps your secrets, costs almost nothing to run, and is nearly as smart as the expensive cloud giants.

Technical Summary: OPENJARVIS – A Decomposed, End-to-End Optimizable Personal AI Stack

1. Problem Statement

Personal AI stacks (e.g., OpenClaw, Hermes Agent) have become central to daily workflows but currently rely almost exclusively on cloud-hosted frontier models. This architecture presents four critical issues:

Privacy: Queries often contain sensitive local data routed to third-party servers.
Cost: Recurring API and subscription costs amount to thousands of dollars annually for heavy users.
Dependency: Users lack ownership of the models and require constant network connectivity.
Energy: Cloud inference consumes orders of magnitude more energy per token than local execution.

While consumer hardware now supports open-weight models (1B–128B parameters) that trail frontier models by only 6–12 months on personal AI tasks, simply substituting a local model into an existing cloud-designed stack fails. Experiments show that replacing a cloud model (e.g., Claude Opus 4.6) with a local one (e.g., Qwen3.5-9B) within frameworks like OpenClaw or Hermes Agent causes a 25–39 percentage point (pp) drop in accuracy.

The root cause is that existing stacks are monolithic: agent prompts, tool descriptions, memory configurations, and runtime settings are co-designed and fused for specific cloud models. Current optimization methods (e.g., prompt optimizers like GEPA or DSPy) can only tune the prompt primitive, closing merely 5 pp of the gap. Optimizing a single primitive in isolation cannot address the misalignment across the entire stack.

2. Methodology: The OPENJARVIS Architecture

To enable local models to compete with cloud systems, the authors propose OPENJARVIS, an architecture that decomposes personal AI systems into five typed, independently editable primitives.

2.1 The Spec Abstraction

OPENJARVIS represents a personal AI system as a typed specification (spec) composed of five primitives:

Intelligence: The language model architecture, weights, and generation parameters.
Engine: The inference runtime, hardware execution path, batching, and quantization settings.
Agents: The reasoning loop, prompts, and tool-use policies.
Tools & Memory: External interfaces, retrieval mechanisms, and persistent user state.
Learning: The optimizer that updates the system from traces (e.g., LoRA, prompt optimizers, or the proposed search algorithm).

This decomposition allows the entire stack to be treated as a single optimizable object, exposing every component as a degree of freedom.

2.2 LLM-Guided Spec Search

The core optimization algorithm is LLM-guided spec search, a local–cloud collaboration that jointly optimizes the four editable primitives (Intelligence, Engine, Agents, Tools & Memory) while using the Learning primitive to drive the process.

Search Time (Cloud): A frontier cloud model (the "Teacher") acts as a reflective proposer. It analyzes execution traces from the current local spec, diagnoses failure clusters (e.g., "student fails on multi-hop calendar lookups"), and proposes coordinated edits across multiple primitives simultaneously (e.g., rewriting a tool description and adjusting a runtime setting).
Inference Time (Local): The resulting optimized spec runs entirely on-device. The cloud model is not involved in inference.
The Gate: To prevent degradation, a "gate" mechanism accepts an edit only if it improves performance on a specific failure cluster without causing unacceptable regressions (defined by a tolerance $\epsilon$ ) on other clusters.
Joint Optimization: Unlike single-primitive optimizers, this method allows a single edit to span multiple primitives, capturing coupled effects (e.g., updating model weights to match a new prompt structure).

3. Key Contributions

OPENJARVIS Architecture: A novel framework that decomposes personal AI into five typed primitives, making the stack end-to-end optimizable and measurable against accuracy, cost, and latency.
LLM-Guided Spec Search: A search algorithm that leverages frontier cloud models to propose edits to local specs at search time, closing the local–cloud accuracy gap by 13–32 pp at 7–11× lower optimization cost than single-primitive baselines.
Pareto-Optimal Local Configurations: Empirical evidence showing that optimized on-device specs can match or exceed cloud accuracy on 4 of 8 benchmarks while reducing marginal API costs by ~800× and end-to-end latency by 4×.
Comprehensive Analysis: An ablation study demonstrating that the gains come from both the proposer (LLM diagnosis vs. evolutionary/random search) and the move space (optimizing all four primitives vs. just one).

4. Experimental Results

The authors evaluated OPENJARVIS across 8 benchmarks (including PinchBench, GAIA, LiveCodeBench, and DeepResearchBench), 11 local models from 4 families, and 7 hardware platforms.

Portability Recovery: Simply swapping a local model into a cloud framework drops accuracy by 24–39 pp. Using the OPENJARVIS spec with the same local model recovers 56–77% of this drop, reducing the residual gap to 5.6–16.5 pp.
Accuracy vs. Cloud: The best single on-device spec (Qwen3.5-122B) achieves 80.3% average accuracy, within 3.2 pp of the best cloud baseline (Claude Opus 4.6 at 83.5%). On 4 of 8 benchmarks, the best local model matches or exceeds cloud performance.
Efficiency Gains: Local configurations offer a ~800× reduction in marginal API cost and a 4× reduction in end-to-end latency compared to cloud baselines.
Search Effectiveness:
- LLM-guided spec search improved local models by 13–32 pp across benchmarks.
- Expanding the editable set from prompts only to all four primitives added 5.5–16.5 pp.
- Using an LLM teacher to diagnose failures added 10.0 pp over evolutionary search at the same move space.
- Accepted edits were distributed across all primitives: Intelligence (16–44%), Agents, Tools, and Engine, confirming that no single component dominates the optimization.

5. Significance and Claims

The paper claims that local personal AI requires more than just model substitution; it requires a portable, measurable, and optimizable stack. OPENJARVIS demonstrates that:

Local-Cloud Parity is Achievable: With the right architecture, on-device systems can approach frontier cloud performance without surrendering local properties (privacy, cost, ownership).
Decomposition is Key: The performance gap is architectural, not just capability-bound. By exposing the assumptions of cloud-designed stacks as editable primitives, local models can be reconfigured to match their strengths.
Hybrid Search is Efficient: Using cloud models only for the search phase (diagnosis and proposal) and running the optimized spec locally at inference time transfers frontier capabilities to local devices without persistent cloud dependency.

The authors emphasize that the resulting local specs make zero cloud calls at inference time. While the search phase incurs a one-time cost (amortized to <$0.001/query after six months of usage), the operational model shifts from a recurring cloud subscription to a one-time optimization followed by local execution.

Limitations noted by the authors:

Statistical precision is limited by 5 independent runs per configuration.
Potential LLM judge bias (using GPT-5-mini) may favor GPT-family outputs.
Evaluation is currently limited to single-machine deployments.
Some cloud baselines showed unexpected low scores due to tool integration issues in the evaluation harness.