🤖 machine learning

Accelerating PayPal's Commerce Agent with Speculative Decoding: An Empirical Study on EAGLE3 with Fine-Tuned Nemotron Models

This empirical study demonstrates that implementing EAGLE3 speculative decoding with a fine-tuned Nemotron model on PayPal's Commerce Agent achieves significant throughput and latency improvements while maintaining output quality, ultimately enabling a 50% reduction in GPU costs compared to standard NVIDIA NIM deployments.

Original authors: Ally Qin, Jian Wan, Sarat Mudunuri, Srinivasan Manoharan

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ally Qin, Jian Wan, Sarat Mudunuri, Srinivasan Manoharan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine PayPal's "Commerce Agent" as a highly skilled, super-fast personal shopper. Its job is to listen to what you type (like "I need a red dress for a wedding under $50") and instantly turn that into a precise list of products to show you.

In the past, this shopper was already very fast because the team taught it specifically about shopping (fine-tuning). But they wanted to make it even faster and cheaper to run, without making it dumber.

This paper is about a clever trick they used called Speculative Decoding, specifically using a method called EAGLE3. Here is how it works, explained with simple analogies.

The Problem: The "One-Step-at-a-Time" Bottleneck

Normally, when an AI writes a sentence, it's like a very careful writer who writes one word at a time.

It thinks of the first word.
It writes it.
It thinks of the second word.
It writes it.
And so on...

This is slow because the computer has to stop and think for every single word. In the world of AI, this is called "autoregressive generation." It's like a chef who tastes the soup, adds a pinch of salt, tastes it again, adds another pinch, tastes it again... instead of just guessing the right amount and adding it all at once.

The Solution: The "Draftsman" and the "Editor"

The team introduced a Speculative Decoding system. Think of it as a two-person team:

The Draftsman (EAGLE3): This is a tiny, super-fast, "lightweight" AI. It's not as smart as the main AI, but it's incredibly quick. Its job is to guess the next few words in the sentence all at once.
The Editor (The Main AI): This is the big, smart, expensive AI. It doesn't write the words itself. Instead, it looks at the Draftsman's guesses and says, "Yes, that's right!" or "No, that's wrong."

How it speeds things up:

Old Way: The Editor writes 1 word, stops, thinks, writes 1 word, stops. (10 words = 10 stops).
New Way: The Draftsman guesses 3 words instantly. The Editor checks all 3 in one single "glance." If the Editor agrees with all 3, they are all accepted at once.
Result: The team gets 3 words for the cost of checking 1. It's like the chef guessing the whole recipe and just checking the final taste, rather than tasting after every pinch of salt.

The Experiment: Testing the Team

The researchers tested this on PayPal's actual shopping system. They set up a race between:

Team A (The Baseline): The standard, powerful AI running on two massive super-computer chips (H100 GPUs).
Team B (The Speculative Team): The same powerful AI, but with the "Draftsman" helper, running on the same two chips.

They tested this under different conditions:

Crowdedness: What happens when 1 person asks a question vs. 32 people asking at once?
Creativity: What if the AI needs to be very strict (Temperature 0) or a little creative (Temperature 0.5)?
Guessing Length: What if the Draftsman guesses 3 words at a time (γ=3) vs. 5 words (γ=5)?

The Big Wins (The Results)

1. The "Sweet Spot" is 3 Words
When the Draftsman guessed 3 words at a time, it was a home run.

Speed: The system became 22% to 49% faster.
Reliability: The Draftsman was right about 35.5% of the time. This is surprisingly stable! Whether the system was busy or quiet, the Draftsman kept hitting that same success rate.
Why not 5? When they tried guessing 5 words, the Draftsman got confused more often (only right 25% of the time). It spent too much time guessing wrong words that the Editor had to reject. It was like a draftsman writing a whole paragraph that was mostly nonsense; the Editor had to waste time crossing it all out.

2. Quality Didn't Drop
A major fear with AI is that "fast" means "dumb." They used another AI to act as a judge to compare the old slow answers with the new fast answers.

Verdict: The answers were identical in quality. The fast method didn't make any mistakes; it just got to the answer faster.

3. The "One Chip" Miracle
This is the most exciting part for saving money.

The "Standard Team" needed two super-chips to handle the load.
The "Speculative Team" could handle the same amount of work using only one super-chip.
Analogy: It's like a delivery company that usually needs two trucks to deliver packages on time. By using a smarter routing system (Speculative Decoding), they realized they could do the exact same job with just one truck, saving 50% on fuel and driver costs.

Why Did This Work So Well?

You might wonder, "Why didn't the Draftsman get right 80% of the time like in other studies?"

Strict Rules: PayPal's AI has to output answers in a very strict format (like a JSON code block with specific brackets and commas). It's harder to guess the next word when you have to follow strict grammar rules compared to just writing a casual story.
Specialized Training: The main AI was trained specifically for shopping. The Draftsman wasn't trained on shopping data, so it was a bit of an outsider.
But it still worked! Even with a lower success rate (35%), the fact that the Draftsman is so cheap and fast meant that even a few correct guesses saved a massive amount of time.

The Bottom Line

PayPal figured out a way to make their shopping assistant significantly faster and cheaper to run without changing the "brain" of the AI.

By adding a "lightning-fast guesser" (EAGLE3) that suggests the next few words for the main AI to check, they cut their wait times in half and could potentially cut their hardware costs in half. It's a perfect example of working smarter, not harder, to get things done.

1. Problem Statement

The deployment of Large Language Models (LLMs) in production e-commerce environments faces a critical trade-off between model quality and inference latency. PayPal's Commerce Agent requires sophisticated reasoning for query formulation, product search, and recommendation. While previous work successfully reduced latency and costs by fine-tuning a small language model (SLM), NEMO-4-PAYPAL (based on llama3.1-nemotron-nano-8B-v1), further optimization was needed to meet strict Service Level Agreements (SLAs) requiring sub-2-second response times without increasing hardware costs.

The core challenge addressed in this paper is: How can inference latency be further reduced and throughput increased for a domain-specific, fine-tuned SLM in a production setting without modifying the model architecture or adding significant hardware resources?

2. Methodology

The authors conducted a comprehensive empirical evaluation comparing a baseline deployment against a speculative decoding implementation.

Target Model: A fine-tuned llama3.1-nemotron-nano-8B-v1 model, optimized for commerce tasks (query understanding, attribute extraction, hypothetical product generation) using LoRA.
Baseline: NVIDIA NIM (containerized inference) deployed on 2× NVIDIA H100 GPUs.
Experimental Setup:
- Technique: Speculative Decoding using EAGLE3, a training-free, draft-model-based method.
- Deployment: vLLM framework on the same 2× NVIDIA H100 hardware to isolate algorithmic effects from hardware differences.
- Configurations: The study spanned 40 distinct experimental configurations varying:
  - Speculative Token Count ( $\gamma$ ): 3 (Spec-3) and 5 (Spec-5).
  - Concurrency Levels: 1, 4, 8, 16, and 32 concurrent requests.
  - Sampling Temperatures: 0 (greedy) and 0.5 (stochastic).
Evaluation Metrics:
- Performance: Throughput (req/s), Mean/Median (P50) Latency, and Speculative Token Acceptance Rate.
- Quality: "LLM-as-Judge" evaluation using pairwise comparisons and individual scoring (relevance, diversity, etc.) to ensure output quality was not degraded.
- Efficiency: GPU utilization analysis.

3. Key Contributions

Production-Scale Empirical Evaluation: The first comprehensive study of speculative decoding applied to a fine-tuned, commerce-specific SLM in a real-world e-commerce environment, covering 40 configurations.
Acceptance Rate Characterization: Discovery that speculative token acceptance rates are remarkably stable (~35.5% for $\gamma=3$ ) across varying concurrency levels and temperatures, simplifying capacity planning.
Quality Preservation Verification: Rigorous validation confirming that speculative decoding produces outputs identical to standard decoding (at temperature 0) or statistically equivalent (at temperature 0.5), with no degradation in quality metrics.
Hardware Efficiency Breakthrough: Demonstration that a single H100 GPU using speculative decoding can match or exceed the performance of two H100 GPUs running the baseline, offering a potential 50% reduction in GPU costs.

4. Key Results

Performance Gains

Throughput: Spec-3 ( $\gamma=3$ ) achieved a 22–49% increase in throughput compared to the NIM baseline. The highest gains (+49%) occurred at low concurrency.
Latency: Spec-3 reduced mean latency by 18–33%. Even at high concurrency (32 requests), a 20% latency improvement was maintained.
Diminishing Returns: Spec-5 ( $\gamma=5$ ) showed significantly lower gains. While it proposed more tokens, the acceptance rate dropped to ~25%, leading to wasted compute on rejected tokens and inferior throughput at higher concurrency.

Acceptance Rate Analysis

Stability: The acceptance rate for Spec-3 remained stable at ~35.5% across all concurrency levels and temperatures.
Temperature Insensitivity: Unlike prior research on general models, temperature changes (0 vs. 0.5) had negligible impact on acceptance rates. The authors attribute this to the guided JSON output format, which constrains the token distribution entropy regardless of temperature.
Comparison to Theory: Observed rates (25–36%) were lower than the 53–82% seen in foundational T5-based studies. This is attributed to:
- Strict schema constraints (JSON) making token prediction harder.
- Distributional shifts from domain-specific fine-tuning.
- Architectural mismatch between the EAGLE3 draft network and the target model.

Hardware Efficiency

Single vs. Dual GPU: At moderate concurrency ( $c \le 8$ ), a single H100 running Spec-3 achieved 3.19 req/s (vs. 3.14 req/s for the dual-H100 baseline) with lower latency (2,292ms vs. 2,529ms).
GPU Utilization: Speculative decoding configurations often showed lower GPU utilization than the baseline despite higher throughput, indicating that requests are completed faster, reducing sustained GPU occupancy.

5. Significance and Implications

Cost Reduction: The study proves that speculative decoding can effectively halve the hardware infrastructure required for a production LLM agent, offering a 50% reduction in GPU costs while maintaining or improving performance.
Optimal Configuration: The paper identifies $\gamma=3$ as the optimal configuration for this workload. It balances the cost of drafting tokens against the probability of acceptance, yielding a theoretical speedup factor (~1.46x) that closely matches empirical results.
Deployment Strategy: The findings suggest a complete optimization pipeline:
1. Fine-tuning (LoRA): Reduces model size and improves task-specific quality.
2. Speculative Decoding (EAGLE3): Accelerates inference at serving time without retraining.
- Combined Impact: These techniques together achieve cumulative latency reductions exceeding 60% compared to the original base model.
Production Readiness: The stability of acceptance rates and the preservation of output quality make this approach highly suitable for immediate production deployment in e-commerce systems.

Conclusion

This paper validates that speculative decoding, specifically using EAGLE3 with a $\gamma=3$ configuration, is a highly effective, low-risk optimization for PayPal's Commerce Agent. It delivers significant throughput and latency improvements on existing hardware, enabling substantial cost savings while strictly adhering to quality SLAs. Future work will focus on fine-tuning the EAGLE3 draft model specifically for commerce data to potentially increase acceptance rates further.