🤖 AI

PANDO: Efficient Multimodal AI Agents via Online Skill Distillation

PANDO introduces an efficient online skill-distillation framework that enables multimodal web agents to improve performance and reduce token consumption by maintaining a structured skill library and employing techniques like progress reflection and cache-aware prompting, achieving a 58.3% success rate on VisualWebArena with significantly lower inference costs compared to existing methods.

Original authors: Yubo Li, Yidi Miao, Haotian Shen, Yuxin Liu

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yubo Li, Yidi Miao, Haotian Shen, Yuxin Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Pay-Per-Click" Trap

Imagine you are hiring a very smart, but expensive, assistant to help you shop online. Every time they look at a webpage, think about what to do, or click a button, you have to pay them a tiny fee (called a "token").

Currently, most AI agents try to get better at their jobs by spending more money. If they fail, they try again. If they get stuck, they ask for a second opinion. They run the same task ten times and pick the best result. This works, but it's like paying for a taxi ride to the grocery store, then paying for a second taxi to go back and check if you forgot your wallet, then a third taxi to double-check the price. It gets the job done, but it's incredibly expensive and slow.

The authors of this paper asked: Can we make an agent that gets cheaper and faster the more it works, instead of just spending more money every time?

The Solution: PANDO (The "Giant Aspen" Agent)

The researchers built an agent named PANDO. They named it after the "Pando" aspen grove in Utah.

The Metaphor: Pando looks like 47,000 separate trees, but they are actually all connected by one single root system underground. When one tree grows, it shares nutrients with the others. When one tree dies, the root system remembers it and keeps the whole forest alive.
The AI Version: PANDO is an AI agent that has a shared "memory root" (a Skill Library). Instead of treating every new task as a brand-new problem, it remembers what it learned from previous tasks and uses those lessons to solve new ones.

How PANDO Works (The 4-Step Loop)

PANDO doesn't just guess and check. It follows a cycle: Plan → Act → Reflect → Learn.

Plan: The "Brain" (a smart, expensive AI) breaks a big task (like "find the cheapest guitar under $500") into small steps.
Act: The "Hands" (a cheaper, faster AI) actually clicks the buttons.
Reflect: A "Manager" checks if the steps worked. Did the price filter actually change the list? If not, why?
Learn (The Magic Part): This is where PANDO gets smarter.
- Skill Library: If the agent successfully finds a cheap guitar, it writes down the recipe: "To find cheap items, click 'Sort by Price' then 'Low to High'." It saves this as a Rule or a Routine.
- Reusing Skills: Next time the agent needs to find the "most expensive" item, it doesn't need to think hard. It just looks at its library, sees the rule, and flips the switch to "High to Low."
- Demotion (The Cleanup): If a rule stops working (e.g., a website changes its layout), the agent marks it as "broken" and stops using it. This prevents the agent from getting stuck in loops trying to use old, broken instructions.

Why It's a Game Changer

The paper tested PANDO on 910 different web tasks (shopping, classifieds, Reddit). Here is what happened:

Success Rate: PANDO succeeded 58.3% of the time. This is better than the current top agents (which were around 54%).
Cost: This is the big win. PANDO used 58% fewer tokens (money) than the next best agent.
The "Free Lunch" Effect:
- Early on: PANDO is a bit slow because it's learning the rules.
- Later on: As it builds its library of skills, it gets faster and cheaper. By the end of the test, it was solving tasks with fewer steps and less money than it started with.
- Analogy: Imagine a student taking a math test.
  - Old Agents: Every time they see a new problem, they re-derive the formula from scratch. It takes forever.
  - PANDO: The first time they see a problem, they solve it and write the formula on a cheat sheet. The next 99 times, they just look at the cheat sheet. They finish the test faster and with less brainpower.

The "Hidden Costs" PANDO Avoids

The paper points out that other agents hide their costs in two ways:

Pre-evaluation Discovery: Some agents spend weeks "training" or "discovering" tools before the test even starts. PANDO learns during the test, so there are no hidden upfront costs.
Rollout Scaling: Some agents just try the task 10 times and pick the winner. PANDO tries it once, but uses its memory to make that one try count.

The Bottom Line

PANDO proves that you don't need to throw more money at an AI to make it better. By giving it a structured memory (a library of skills) and a way to clean up bad memories (demotion), the agent becomes more efficient the more it works.

It's the difference between a worker who forgets everything every day and has to be retrained constantly, versus a worker who keeps a notebook of "how to do things" and gets faster every single day.

Key Takeaway: PANDO doesn't just get smarter; it gets cheaper to run as it gains experience.

Technical Summary: PANDO – Efficient Multimodal AI Agents via Online Skill Distillation

Problem Statement

Recent advancements in multimodal web agents have largely relied on a "token economy" where improved performance is purchased through increased inference costs. This includes strategies such as expanding rollout searches (e.g., Best-of-N), adding verifier passes, conducting offline tool discovery, or stacking specialist models. While effective for benchmark scores, this approach creates a linear or super-linear relationship between token expenditure and success, leading to high latency, energy consumption, and hidden costs associated with pre-evaluation discovery.

The authors identify three specific mechanistic inefficiencies in current agent trajectories on the VisualWebArena (VWA) benchmark:

Repeat-action loops: Agents frequently get stuck in loops of repeated actions (34–42% of image-annotated failures).
Hidden discovery costs: Systems like WALT incur significant offline exploration costs that are often excluded from reported per-task metrics.
Prompt-architecture inefficiency: Text/caption methods exhibit low prompt-cache utilization (<11%), failing to leverage stable prefixes as the agent accumulates experience.

The central question posed is whether an agent can become more efficient as it accumulates experience, rather than paying a fixed "reasoning tax" on every task.

Methodology: The PANDO Framework

PANDO (named after the Pando aspen grove, a single root system supporting many trunks) is a single-rollout, online skill-distillation framework. It operates on a Plan → Act → Reflect → Learn loop, designed to grow a structured Skill Library during evaluation without requiring a pre-evaluation discovery budget.

Core Components

Structured Skill Library ( $S_t$ ):
- The library is a disjoint union of Rules (deterministic, pattern-triggered guardrails) and Routines (parameterized, executable program-as-action skills).
- Retrieval: Unlike embedding-based retrieval, PANDO uses literal keyword containment for deterministic, auditable, and cache-friendly retrieval.
- Lifecycle Management: The library is not static; it grows online. Successful sub-trajectories are induced into routines, while failed ones are tracked.
Learning and Distillation:
- Induction: After a successful trajectory, a Learning Module extracts candidate skills if they possess a reusable subgoal template and verified selector patterns.
- Polarity-Pair Merging: To prevent library bloat, routines that differ only by direction (e.g., "cheapest" vs. "most expensive") are merged into a single parameterized routine $f_{\pm}(x, d)$ .
- Confidence-Based Demotion: Skills maintain running confidence estimates ( $c_s = \alpha_s / (\alpha_s + \beta_s)$ ). If a skill fails repeatedly, it is moved to a demotion blacklist, preventing the agent from rediscovering and re-failing with brittle routines.
Execution Economy:
- Hierarchical Routing: A strong model (Planner) is used sparsely for decomposition and reflection. A cheaper model (Actor) handles high-frequency grounding.
- Visual Compression: Reduces the token load of the dominant Actor term by compressing visual inputs.
- Cache-Aware Prompting: The prompt layout is structured to place stable instructions and skill summaries before volatile observations, maximizing KV-cache utilization as the library grows.
Cost Decomposition:
The paper formalizes agent cost into four additive terms:
$C(\pi; B) = \frac{C_{pre}}{|B|} + N_{rollout} C_{exec} + C_{verify} + C_{induce}$
PANDO aims for $C_{pre}=0$ (no offline discovery), $N_{rollout}=1$ (single rollout), and a small $C_{induce}$ paid strictly within the task stream, while minimizing $C_{exec}$ through skill reuse.

Key Contributions

Token-Economics Framing: The authors formalize a cost identity that separates pre-evaluation discovery, rollout scaling, verification, and online induction, arguing that current benchmarks often obscure these distinct cost currencies.
Structured Skill-Learning Framework: PANDO integrates pattern-indexed rules, parameterized routines, online distillation, polarity-pair merging, confidence-based demotion, and cache-aware prompting into a single agent.
Intrinsic Efficiency Metrics: Beyond Success Rate (SR), the paper introduces Action Repetition Rate (ARR), Step Overhead Ratio (SOR), and Prompt Cache Utilization to diagnose trajectory-level inefficiencies.
State-of-the-Art Results: PANDO achieves a new Pareto frontier on VWA, delivering higher success rates with significantly fewer tokens than all evaluated baselines.

Experimental Results

Evaluated on all 910 tasks of the VisualWebArena benchmark:

Success Rate (SR): PANDO achieves 58.3%, surpassing SGV (54.0%) and a reproduction of WALT (45.2%).
Token Efficiency: PANDO uses 115K tokens per task on average. This is 58% fewer than SGV (275K) and 61% fewer than WALT (294K).
Intrinsic Metrics:
- ARR: 9.1% (lowest among automated methods), indicating effective loop avoidance.
- SOR: 1.8× (lowest), showing that failed tasks do not incur excessive step penalties.
- Cache Utilization: 72.4% (highest), demonstrating the effectiveness of stable prompt prefixes.
Ablation Studies (VWA-300):
- Skill Components: Rules, routines, and online distillation provided the majority of the SR lift (from 38.6% to 57.3%).
- Efficiency Components: Hierarchical routing, visual compression, and cache-aware prompting contributed minimally to SR but were responsible for the majority of token reduction (from 147K to 117K) and increased cache utilization.
Learning Dynamics: The system exhibits a "cold-start" phase with high step counts (~30 steps/task) that stabilizes to ~8.5 steps/task as the library matures. The skill library grew from 12 seed routines to 47 induced routines (32 active after demotion) over the stream.

Significance and Claims

The paper claims that PANDO demonstrates that web-agent progress need not be purchased solely through increased rollouts, hidden discovery budgets, or per-step specialist stacking.

Reusability as Capital: The core insight is that past token expenditure should become reusable capital. By distilling experience into a structured, auditable library, the agent lowers its marginal inference load over time.
Transparency: Unlike black-box vector stores or offline discovery phases, PANDO's skill library is inspectable, deterministic, and stable under prompt caching.
Benchmark Reporting: The authors argue that computer-use benchmarks must report SR alongside raw tokens, cached-token share, hidden discovery spend, and latency to provide a complete picture of agent efficiency.

The paper concludes that PANDO achieves a new Pareto-optimal point: higher accuracy with lower marginal cost, suggesting that "efficiency is not obtained by doing less, but by ensuring that what is done has somewhere to go."