💬 NLP

Scalable Behaviour Cloning on Browser Using via Skill Distillation

This paper proposes a scalable approach to training browser agents by distilling human interaction trajectories into compact, retrievable natural-language skills organized within a skill graph, thereby leveraging existing user data to overcome decision-making bottlenecks rather than relying on manually designed tasks.

Original authors: Kaisen Yang, Zheng Jiang, Yuzhao Peng, Houde Qian, Boshi Zhang, Youjie Zheng, Shijin Hong, Qingle Liu, Ruoyu Han, Bohan Lyu, Bingxiang He, Eren Cai, Calvin Xiao, Qinhuai Na

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kaisen Yang, Zheng Jiang, Yuzhao Peng, Houde Qian, Boshi Zhang, Youjie Zheng, Shijin Hong, Qingle Liu, Ruoyu Han, Bohan Lyu, Bingxiang He, Eren Cai, Calvin Xiao, Qinhuai Na

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to use a web browser. You could show the robot a video of a human clicking "Submit" on a specific form, and the robot might learn to click that exact spot. But the moment the website changes its layout, or the button moves two inches to the left, the robot fails. It's like teaching someone to drive by memorizing the exact position of the steering wheel in one specific car; if they get into a different car, they crash.

This paper, BrowserBC, argues that the problem isn't that robots can't click buttons or type letters. The problem is that they don't know what to do next when they are lost. They lack "decision priors"—the gut feeling or common sense that humans have about how to navigate a website.

Here is how the authors solve this, using simple analogies:

1. The Problem: The Robot is "Amnesiac"

Current AI agents are like tourists who have never been to a city before. They can see the street signs (the webpage) and they can walk (click and type), but they don't know the shortcuts, the dead ends, or how to recognize when they've actually arrived at their destination. They end up wandering in circles, trying random things until they get lucky.

The authors say we have a massive, untapped resource to fix this: the billions of hours humans have already spent browsing the internet. Every time a human successfully fills out a form or finds a product, they leave behind a "trace" of their journey.

2. The Solution: Turning "Footprints" into "Instruction Manuals"

The paper proposes a system called Skill Distillation. Instead of saving the raw video of a human clicking buttons (which is messy and specific to one moment), the system converts those actions into Natural Language Skills.

Think of it this way:

Raw Trace: A video of a person walking from their house to the grocery store, stopping at a red light, tripping on a crack, and buying milk. This is too specific. If the road is closed tomorrow, the video is useless.
BrowserBC Skill: A written recipe card that says: "To buy milk: Go to the store, find the dairy aisle, grab a carton, and check the expiration date. If the store is closed, go to the next one."

This "recipe card" is a Skill Card. It doesn't care about the specific street names or the color of the traffic light. It cares about the logic of the task.

3. The Library: A "Skill Graph"

If you just collected thousands of these recipe cards, you'd end up with a messy pile where some say "Buy milk" and others say "Go to the dairy section," causing confusion.

The authors organize these cards into a Skill Graph. Imagine a well-organized library or a flowchart:

If you need to "Fill a Form," the system looks at the graph and finds the general "Fill Form" card.
If that form is for a "Payment," it links to a more specific "Payment Form" card.
If the form fails, it links to a "Recovery" card that says, "If you get an error, check your email address."

This allows the system to grow smart without getting cluttered. It consolidates knowledge rather than just hoarding it.

4. The Magic Trick: "Distill Once, Use Cheaply"

This is the most powerful part of the paper.

The Distiller: A very smart, expensive AI (like a PhD student) watches a human browse a website. It writes the perfect "Skill Card" (the recipe).
The Executor: A smaller, cheaper, faster AI (like a high school intern) is then given that card to do the actual work.

The "PhD student" does the hard thinking once to create the rulebook. The "intern" just follows the rulebook. This means you don't need a super-computer for every single task. You can record human expertise once and let thousands of smaller, cheaper robots use that knowledge forever.

5. The Results: Less Wandering, More Success

The authors tested this on real websites and found that agents using these "Skill Cards":

Solved more tasks: They succeeded about 20% to 35% more often than agents without the cards.
Took fewer steps: They didn't wander around as much. They knew where to go.
Recovered from mistakes: When things went wrong, they knew how to fix it because the "Recovery" part of the skill card told them what to do.
Worked on new websites: Because the skills were written in general language (not specific coordinates), they worked even when the websites changed their design.

Summary

The paper argues that the future of smart web agents isn't about making them click faster or see better. It's about giving them a library of human wisdom. By turning messy human browsing history into clean, reusable "instruction manuals," we can teach robots how to navigate the internet with the same common sense that humans use, without needing to relearn everything from scratch every time.

Technical Summary: Scalable Behaviour Cloning on Browser Using Skill Distillation

1. Problem Statement

The paper identifies a critical bottleneck in contemporary web agents: while modern models have largely solved low-level browser operations (parsing DOMs, executing clicks, and keystrokes), they remain inefficient on real-world websites due to decision-making under incomplete information. Agents observe only the current page state, lacking global knowledge of site structure, reliable entry points, path costs, and completion signals. Consequently, capable agents often resort to costly trial-and-error exploration.

The authors argue that the necessary decision priors are abundant in human behavior but are currently under-exploited. Raw human browser traces are poor units for direct supervision because they are long, noisy, redundant, and tightly coupled to volatile page states (e.g., specific coordinates, transient DOM IDs, login states). Directly cloning these low-level actions results in brittle, page-specific behaviors that fail to transfer. The core problem is thus how to convert these massive, noisy human interaction logs into reusable, transferable procedural knowledge that agents can apply to unfamiliar websites.

2. Methodology: BrowserBC

The authors propose BrowserBC, a scalable framework that converts browser interaction traces into a retrievable, composable, and executable library of natural-language skills via skill distillation. The framework operates through four distinct stages:

3.1 Behavioral Evidence Abstraction

Raw trajectories ( $\tau$ ) are first cleaned and semantically segmented. The system filters out non-contributory interactions (misclicks, idle waits) and volatile artifacts (coordinates, specific selectors). The remaining data is converted into structured evidence units ( $u$ ), which capture:

The task instruction ( $x$ ).
Context summaries before and after the segment ( $c_{pre}, c_{post}$ ).
A summary of the demonstrated behavior ( $b$ ).
Observed feedback ( $f$ ) and outcomes ( $y$ ).
This stage lifts the data from page-specific action traces to generalizable procedural evidence.

3.2 Procedural Skill Distillation

Evidence units are distilled into structured natural-language skill cards ( $\hat{s}$ ). A skill is defined not as a replay script, but as a transferable procedure containing specific fields:

Intent: What the skill accomplishes.
Applicability: When and where it applies (preconditions).
Execution Steps: Generalized procedures (e.g., "identify fields by semantic labels" rather than "click coordinate X,Y").
Progress Milestones & Termination: How to verify progress and when to stop.
Failure Recovery: Strategies for handling common errors.
Negative Constraints: What not to do.

Crucially, the distillation process explicitly strips away session-specific details (coordinates, login states) to ensure the skill is model-agnostic and transferable.

3.3 Skill-Library Construction (Skill Graph)

To prevent unbounded accumulation and redundancy, the system organizes skills into a Skill Graph ( $G$ ). When a new skill is generated, the system decides whether to:

Add it as a new entry.
Merge it with an existing compatible skill.
Register it as a specialization of a general skill.
Edges in the graph encode relationships such as temporal dependency, specialization, alternative procedures, or incompatibility. This allows the library to grow through consolidation rather than simple accumulation, keeping retrieval and updates localized.

3.4 Skill-Conditioned Execution

At inference time, a lightweight retriever selects a compact subset of relevant skills ( $R_t$ ) based on the current task and browser context. These skills are inserted into the agent's context as natural-language guidance. The agent then grounds these high-level instructions into executable actions on the live page. The skills act as calibrated priors that bias the agent's decision-making but do not override the agent's ability to adapt to the current state.

3. Key Contributions

Reframing the Bottleneck: The paper redefines the inefficiency of web agents as a decision-making problem under incomplete information, identifying human traces as a source of large-scale decision data where the reusable unit is a transferable natural-language skill, not a click or full trajectory.
BrowserBC Framework: A scalable system that distills traces into structured skill cards, consolidates them into a skill graph, and decouples skill provenance from execution. This allows experience recorded once (by a capable model or human) to be reused by smaller, cheaper models.
Comprehensive Evaluation: The authors evaluate BrowserBC across three benchmarks (WebArena-Hard, ClawBench, and OSWorld-style desktop tasks), measuring success rates, interaction efficiency, failure recovery, and cross-substrate generalization.

4. Experimental Results

The framework was evaluated on three benchmarks:

WebArena-Hard (Controlled): BrowserBC improved overall task success from 60.5% to 81.4% (+20.9 points). The gains were most significant in multi-site and shopping tasks (+31.2% and +28.6% respectively), where procedural uncertainty is high. Interaction steps were reduced by 27.3% on average.
ClawBench (Live Production): On live websites with volatile layouts, the baseline solved 32.9% of tasks, while BrowserBC solved 68.4% (+35.5 points). This significant gain on dynamic, unsaturated environments demonstrates that the skills encode transferable procedural knowledge rather than memorized page states.
OSWorld (Cross-Substrate): In a diagnostic study on desktop tasks, skills enabled the agent to solve 6 additional cases that were previously unsolvable, primarily by supplying missing task schemas and terminal evidence.

Decoupling Distillation and Execution:
Experiments showed that skills distilled by a stronger model (Claude-Sonnet-4.6) significantly improved the performance of weaker executing agents. A smaller agent equipped with Sonnet-distilled skills achieved 77% success, approaching the 80% achieved by the large agent itself, validating the "distill once, reuse cheaply" premise.

5. Significance and Claims

The paper claims that the scalability of browser agents may depend less on manually designed tasks or scaling up individual agent models, and more on leveraging the collective skills already expressed by internet users.

By converting raw, noisy traces into auditable, natural-language skill cards, BrowserBC offers a path to accumulate and reuse experience across different agents and environments. The authors emphasize that their approach is orthogonal to improvements in environments or rewards; instead, it treats existing human interaction data as a scalable supervision signal. The work suggests that procedural priors (knowing what to do and how to verify it) are more valuable than low-level grounding capabilities for solving complex, long-horizon web tasks.

The paper concludes modestly, noting that while skills improve success and efficiency, they are not a substitute for perfect low-level execution or shortest-path policies, and their benefit is bounded by execution reliability and retrieval correctness.