🤖 AI

A/B Agent: A Self-Evolving Agent for Strategy Iteration in Industrial A/B Testing

This paper introduces A/B Agent, a self-evolving framework that leverages a hierarchical experience tree and multi-path Tree-RAG to autonomously generate, execute, and iteratively refine industrial recommendation strategies through closed-loop A/B testing, achieving significant GMV improvements in real-world e-commerce systems.

Original authors: Zhuohang Jiang, Yuxin Chen, Yongsen Pan, Zheng Hu, Wenqi Fan, Qing Li, Hongyang Wang, Jun Wang, Wenwu Ou

Published 2026-08-06

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhuohang Jiang, Yuxin Chen, Yongsen Pan, Zheng Hu, Wenqi Fan, Qing Li, Hongyang Wang, Jun Wang, Wenwu Ou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive, high-speed spaceship (a recommendation system) trying to deliver the perfect snack to every passenger. Sometimes, you guess right, and everyone is happy. Other times, you hand out a spicy chip to someone who wanted something sweet, or you run out of fuel because you didn't plan the route well. In the real world, companies like Kuaishou use a method called "A/B testing" to figure out the best route. It's like running two different versions of the ship at the same time: one with the old map, one with a new idea. If the new map gets more passengers to their snacks faster, you keep it. But here's the catch: doing this manually is exhausting. It requires a team of expert navigators to constantly draw new maps, run tests, check the results, and tweak the settings. It's slow, and they often forget the lessons they learned from yesterday's failed detours.

Recently, scientists have been teaching computers to read and understand these old maps using "Large Language Models" (super-smart AI that reads text) and "RAG" (Retrieval-Augmented Generation, which is like giving the AI a library to look up answers before it speaks). But even these smart AI helpers have a problem: they often treat the library like a flat pile of papers. They might find a story about a "desert route" when you actually need a "mountain route," or they might miss the fact that a strategy that worked for "snacks" might be dangerous for "drinks." They struggle to see the big picture of how different parts of the ship connect, and they can't easily learn from their own mistakes over time without a human boss telling them what to do next.

This is where the paper introduces A/B Agent, a new kind of "self-evolving" AI navigator designed to fix these problems. Instead of just reading a flat pile of notes, A/B Agent builds a giant, organized "Experience Tree." Imagine a tree where the roots are the big business goals, the branches are different parts of the ship (like the snack bar or the engine room), and the leaves are specific strategies that worked or failed in the past. When the AI needs a new idea, it doesn't just search for keywords; it climbs the tree to find the exact branch that matches the current situation, ensuring it grabs the right kind of knowledge.

Once it picks a strategy, A/B Agent doesn't just stop there. It acts like a tireless scientist who runs the test, watches the results, and then immediately updates its own tree. If a new route gets more passengers but makes the engine overheat (a "guardrail" metric), the AI notices, tweaks the settings, and tries again. It keeps doing this loop—thinking, testing, learning, and updating its own knowledge base—until it finds the perfect balance. The paper shows that this system isn't just a theory; when tested in a real-world short-video shopping environment, it successfully improved the "Gross Merchandise Value" (the total money spent by users) by 4.829% while keeping all the safety metrics positive. It even outperformed some of the world's most advanced AI models in creating strategies that were both correct and creative. Essentially, A/B Agent turns the chaotic process of trial-and-error into a smart, self-improving cycle that keeps getting better on its own.

Technical Summary: A/B Agent for Industrial Strategy Iteration

1. Problem Statement

Industrial recommendation systems rely heavily on large-scale A/B testing to iterate strategies, a process that is currently labor-intensive and time-consuming. It requires human experts to repeatedly design strategies, configure experiments, analyze results, and adjust parameters. A critical bottleneck is the fragmentation of historical knowledge; valuable insights from past experiments are scattered across heterogeneous sources (documents, code, logs) without a unified abstraction, making systematic reuse difficult.

While Retrieval-Augmented Generation (RAG) agents have been proposed to assist, existing approaches suffer from two main limitations:

Flat Knowledge Organization: Traditional RAG systems retrieve text chunks based on semantic similarity, failing to model the hierarchical relationships between business scenarios, recommendation stages, optimization objectives, and experimental contexts. This leads to mismatched retrieval and poor cross-scenario transfer.
Lack of Self-Evolution: Current agents typically provide one-shot recommendations without a long-horizon optimization framework. They cannot autonomously refine strategies based on sequential A/B feedback or progressively consolidate successful practices into a knowledge base.

Consequently, the industry lacks an automated system capable of transforming fragmented historical records into structured, transferable experiences and continuously refining strategies through online feedback loops.

2. Methodology

The paper proposes A/B Agent, a closed-loop framework designed for industrial recommendation strategy optimization. The framework consists of three tightly coupled components:

2.1 Historical Strategy Knowledge Organization

To address knowledge fragmentation, the system structures historical experiment records into a Hierarchical Experience Tree.

Record Structuring: Raw experiment reports are converted into a unified schema covering business background, scenario, stage, objective, strategy description, parameters, metrics, and risks.
Agentic Extraction: A strategy agent decomposes records into atomic, self-contained "strategy chunks" from multiple perspectives (problem, scenario, mechanism).
Tree Construction: These chunks are organized into a tree $T_E = (V_E, E_E)$ where nodes represent categories (domain, scenario, stage, objective) and leaves store specific strategy details. This structure allows for precise retrieval within matched branches and knowledge transfer across related branches.

2.2 Autonomous Target-Aware Strategy Generation

Given a new optimization request, the agent generates executable strategies using a Multi-Path Tree-RAG approach:

Multi-Path Retrieval: The system maps the request to a semantic path in the experience tree and retrieves candidates using both sparse (exact terms) and dense (semantic) retrieval.
Context-Aware Boosting: Unlike flat RAG, this module boosts candidates that structurally match the request's scenario, stage, and objective. It calculates a tree-path relevance score based on the shortest-path distance between nodes in the hierarchy.
Reranking and Generation: A reranker (Qwen-Reranker) performs fine-grained scoring based on mechanism and constraints. The strategy generator then adapts transferable mechanisms to the target task, considering engineering constraints and deployment conditions.
Judging: Generated candidates are evaluated by agent-based judges for contextual relevance, evidence consistency, and safety before being converted into an executable A/B test configuration.

2.3 Experiment-Guided Strategy Self-Evolution

Post-deployment, the system enters a self-evolution loop:

Experiment Tree Construction: Successive experiment versions are organized into an Experiment Tree ( $T^*_A$ ), where nodes represent strategy variants and edges represent modifications.
Utility Evaluation: The system defines a utility function $U(v)$ that balances core metric gains against guardrail metric degradation (penalizing negative impacts on user experience or platform stability).
Iterative Tuning: The agent compares parent-child and sibling nodes to identify effective modifications. If a branch saturates or violates guardrails, it retrieves alternative mechanisms from the experience tree to create new branches. Validated outcomes are distilled back into the experience tree, closing the loop for continuous knowledge accumulation.

3. Key Contributions

The paper outlines the following contributions:

A/B Agent Framework: A closed-loop agent for mining historical strategies, enabling cross-scenario transfer, analyzing A/B results, and optimizing parameters in industrial recommender systems.
Hierarchical Experience Tree: A novel structure that decomposes historical strategies into transferable skills, supporting structured, target-aware retrieval and generation.
Self-Evolution Mechanism: An experiment-tree-guided approach for A/B tuning that enables joint metric analysis, parameter optimization, and continuous knowledge updates without manual intervention.
Industrial Dataset: The construction of a dataset containing 310 historical recommendation strategies across three e-commerce scenarios, including configurations, outcomes, and multidimensional metrics.
Empirical Validation: Extensive offline and online evaluations demonstrating the framework's effectiveness.

4. Experimental Results

The paper evaluates A/B Agent across three industrial e-commerce scenarios on the Kuaishou platform.

Offline Performance:
- A/B Agent achieved the highest average overall score of 7.244, outperforming the strongest general-purpose LLM (Claude-Sonnet-4.6, 7.151) by 1.3%.
- Compared to the strongest RAG baseline in each scenario, A/B Agent improved the overall score by 25.0%, 31.7%, and 23.5%, respectively.
- It surpassed GPT-5.5 in correctness and Claude-Sonnet-4.6 in novelty, demonstrating that structured industrial knowledge and transferability-aware retrieval yield superior results.
Online A/B Testing:
- In a real-world deployment for short-video e-commerce recommendations, A/B Agent achieved a 4.829% improvement in Cart GMV.
- Crucially, it maintained positive gains across all guardrail metrics (including Live GMV, Watch Time, and Platform Orders), whereas earlier experimental iterations by the agent showed trade-offs that were subsequently resolved through self-evolution.
- The agent successfully navigated the Pareto frontier, moving from initial configurations with guardrail risks to a final state that improved both business metrics and platform safety.
Ablation Study:
- Removing the knowledge base caused an absolute score decline of 0.238 (approximately 3.3% of the total score), highlighting the importance of historical experience.
- Replacing the hierarchical Tree-RAG with Flat RAG resulted in an absolute decline of 0.015 (approximately 0.2% of the total), confirming the value of hierarchical organization.
- Removing retrieval enhancements (e.g., scenario-aware boosting) led to absolute declines ranging from 0.070 to 0.122 (approximately 1.0% to 1.7% of the total).

5. Significance and Claims

The paper claims that A/B Agent addresses the gap between general LLM agents and the specific, complex requirements of industrial strategy iteration. Its significance lies in:

Systematizing Fragmented Knowledge: By organizing unstructured historical data into a hierarchical tree, the system enables reliable knowledge transfer across different business contexts, which flat RAG systems fail to achieve.
Autonomous Continuous Improvement: The framework moves beyond one-shot generation to a self-evolving loop where the agent learns from online feedback, refining strategies and parameters autonomously.
Balancing Optimization and Safety: The experiment-guided self-evolution mechanism explicitly models the trade-off between core metric gains and guardrail constraints, allowing the agent to navigate toward Pareto-optimal solutions that human experts might struggle to find manually.

The authors conclude that A/B Agent effectively translates historical strategy knowledge into measurable online business improvements, offering a scalable solution for industrial strategy optimization while maintaining system health. Future work is noted to focus on stronger requirement alignment and automated engineering-feasibility verification.