Technical Summary: Generative AI and Sales Productivity: Field Experiments in Online Retail

1. Problem and Research Question

The rapid diffusion of Generative Artificial Intelligence (GenAI) has sparked interest in its potential to improve productivity across the economy. However, while existing literature provides causal evidence that GenAI enhances worker-level task efficiency (e.g., in coding, customer support, and legal analysis), there is limited causal evidence regarding its impact on sales performance in real-world, consumer-facing business workflows.

This paper addresses the question: Does GenAI adoption improve sales productivity in online retail, and if so, through what mechanisms and for which market participants? The authors investigate this within the context of a leading cross-border e-commerce platform, where reducing search, information, and communication frictions is critical to the customer journey.

2. Methodology and Experimental Design

The study employs a series of seven large-scale randomized field experiments conducted between September 2023 and June 2024 on a single global cross-border e-commerce platform. The experiments cover millions of users and products, allowing for causal identification of GenAI's impact.

Experimental Setup

• Scope: Seven distinct business workflows were re-engineered with GenAI, spanning three functional areas:
  1. Consumer and Seller Services: Pre-sale Service Chatbot, Chargeback Defense, Live Chat Translation.
  2. Consumer-Product Matching: Search Query Refinement, Product Description Generation.
  3. Advertising and Promotion: Marketing Push Message Creation, Google Advertising Title Optimization.
• Design: In each workflow, a treatment group was exposed to the GenAI-enhanced process, while a control group continued with the platform's standard baseline practices (e.g., pre-programmed auto-responses, human-generated content, or standard lexical translation).
• Randomization: Randomization occurred at the consumer level for six experiments and at the product level for one (Google Advertising Title). Consumer overlap across experiments was minimal (<1%).
• Data: Granular consumer- and product-level data (views, clicks, orders, expenditure, returns, ratings) were available for five of the seven experiments. For the remaining two (Chargeback Defense and Live Chat Translation), the authors relied on internal platform metrics.
• Estimation: The authors use Ordinary Least Squares (OLS) regression with cohort fixed effects to estimate the Average Treatment Effect (ATE) on sales (total consumer expenditure). They also analyze conversion rates, cart value, and post-purchase outcomes (returns and ratings).

3. Key Contributions

This paper makes three primary contributions to the literature:

1. Demand-Side Evidence: Unlike prior studies focusing on supply-side worker efficiency (time savings, task completion), this study provides large-scale causal evidence on demand-side business outcomes (sales, conversion, consumer experience).
2. Multi-Workflow Comparison: By conducting experiments within a single firm under a common organizational environment, the study isolates the marginal contribution of GenAI across different workflows, controlling for implementation variance.
3. Heterogeneity Analysis: The paper documents substantial heterogeneity in treatment effects, specifically identifying that less experienced consumers benefit disproportionately more from GenAI interventions.

4. Key Results

A. Sales Impact Across Workflows

GenAI adoption increased sales in most workflows, though the magnitude varied significantly based on the baseline conditions:

• Pre-sale Service Chatbot: The largest effect, with a 16.3% increase in sales. The control group received no live pre-sale support, while the treatment group received 24/7 GenAI assistance.
• Search Query Refinement: A 2.9% increase in sales by improving semantic understanding and translation of multilingual queries.
• Product Description: A 2.1% increase in sales by generating comprehensive, structured text for products that previously lacked descriptions.
• Marketing Push Message: A positive but statistically insignificant point estimate (1.6%), though significant increases in conversion rates were observed.
• Google Advertising Title: A statistically insignificant negative effect (-4.5%). The authors attribute this to the GenAI model not being fine-tuned for the specific advertising context, leading to lower quality scores and visibility.
• Operational Metrics: Chargeback Defense saw a 15% increase in defense success rates, and Live Chat Translation improved consumer satisfaction by 5.2%.

B. Mechanisms: Conversion vs. Cart Value

The sales gains are driven primarily by increased purchase incidence (conversion rates) rather than increased spending intensity (cart value).

• Conversion Rates: Increased by 1% to 22% across workflows.
• Cart Value: Remained largely unchanged.
• Post-Purchase Outcomes: There is no evidence that sales gains came at the expense of consumer satisfaction. Product return rates and customer ratings did not deteriorate; in some cases (Product Description, Marketing Push), return rates significantly declined.
• Interpretation: These patterns suggest GenAI improves the shopping experience by reducing search, information, and communication frictions, rather than by persuading consumers to spend more per item.

C. Aggregate Economic Value

Aggregating the four workflows with positive sales effects, the authors estimate an annual incremental value of approximately $4.60 to $5.20 per consumer. This represents roughly 5.5–6.2% of the per-user revenue growth observed in global e-commerce between 2023 and 2024.

D. Heterogeneous Treatment Effects

• Consumers: Less experienced consumers (lower past spending, fewer login days, shorter registration tenure) benefit disproportionately more from GenAI workflows. This suggests GenAI is particularly effective at reducing frictions for users with lower baseline capabilities.
• Sellers: Point estimates suggest larger gains for smaller and newer sellers, though these results are less statistically precise.
• Products: Heterogeneity across product groups is mixed and context-dependent, with no uniform pattern observed.

5. Significance and Claims

The paper claims to provide novel, large-scale causal evidence on how GenAI shapes sales productivity in online retail. Its significance lies in:

• Demonstrating Immediate Value: Showing that GenAI can generate economically meaningful short-term gains in mature retail settings, even before long-term organizational adaptations occur.
• Identifying Friction Reduction: Highlighting that the primary mechanism for value creation is the reduction of market frictions (search, information, communication) rather than purely persuasive effects.
• Equity Implications: Suggesting that GenAI adoption may have distributional effects, disproportionately benefiting less experienced consumers who face higher initial barriers to entry in digital marketplaces.

The authors remain modest regarding long-term implications, noting that their estimates reflect short-run effects and do not account for general equilibrium forces, competitive responses, or full return on investment (ROI) calculations involving capital and labor costs. They conclude that while GenAI offers immediate value, its long-run impact will depend on how firms integrate these tools into broader organizational structures and how market competition evolves.