🤖 AI

Enhancement of E-commerce Sponsored Search Relevancy with LLM

This paper presents a novel e-commerce sponsored search relevance model that fine-tunes the LLAMA2 7B language model using Low-Rank Adaptation (LoRA) to achieve 89.43% accuracy in classifying ad-query relevance, outperforming baseline and GPT-4 models while enhancing operational efficiency and privacy.

Original authors: Md Omar Faruk Rokon, Andrei Simion, Weizhi Du, Musen Wen, Hong Yao, Kuang-chih Lee

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Md Omar Faruk Rokon, Andrei Simion, Weizhi Du, Musen Wen, Hong Yao, Kuang-chih Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a massive, chaotic warehouse (like Walmart.com) looking for a specific item, say, "waterproof hiking boots." You shout your request, and the store's digital assistant needs to instantly find the perfect pair of boots to show you.

In the past, this assistant was like a robot that only understood exact word matches. If you said "rain boots," it might get confused if the product was listed as "waterproof hiking shoes." It often missed the nuance of what you actually wanted, showing you things that were close but not quite right.

This paper describes how the team at Walmart built a much smarter assistant using a type of artificial intelligence called a Large Language Model (LLM), specifically a version called LLaMA2. Here is the simple breakdown of what they did and why it matters:

1. The Problem: The "Needle in a Haystack"

In e-commerce, there are millions of products and millions of ways people search for them. The challenge is matching a user's short, sometimes vague search query (like "waterproof hiking boots") with the right product title from a massive list.

The old way: Used rigid rules. It often failed to understand that "hiking boots" and "outdoor shoes" might mean the same thing to a human.
The new way: Use an AI that "reads" and "understands" the meaning behind the words, just like a human does.

2. The Solution: A "Specialized Intern" (LoRA)

The team took a very powerful, pre-trained AI brain (LLaMA2 7B) that already knows a lot about human language. However, this brain is huge and expensive to run. They didn't want to retrain the whole brain from scratch (which would be like teaching a genius to read a dictionary all over again).

Instead, they used a technique called LoRA (Low-Rank Adaptation).

The Analogy: Imagine the LLaMA2 model is a brilliant, general-purpose librarian who knows everything about books, movies, and science. You want them to work specifically in the "Hiking Gear" section of the library.
The Trick: Instead of firing the librarian and hiring a new one, or making the librarian re-read the entire library, you give them a specialized cheat sheet (the LoRA adapter). This cheat sheet teaches them the specific rules of "hiking gear" without changing their entire personality or memory.
The Result: The librarian becomes an expert in hiking gear instantly, using very little extra energy or time.

3. The Job: The "Relevance Judge"

The team taught this specialized AI to act as a judge for search results. When a user types a query and the system suggests a product, the AI looks at the pair and gives them a score in one of three categories:

Relevant: "Yes, these are exactly what the user wants." (e.g., Query: "Waterproof boots" / Product: "Gore-Tex Hiking Boots").
Partially Relevant: "It's close, but maybe not perfect." (e.g., Query: "Hiking boots" / Product: "Waterproof rain boots" – good for rain, but maybe not for hiking).
Irrelevant: "No, this is the wrong item entirely." (e.g., Query: "Hiking boots" / Product: "Silicone baking mats").

4. The Results: Smarter and Faster

The team tested this new "Specialized Intern" against other smart models, including a very famous one called GPT-4 and older, standard search tools.

Accuracy: The new model got it right 89.43% of the time.
Comparison: It beat the older standard models and even outperformed GPT-4 in this specific task.
Why GPT-4 lost: The paper suggests GPT-4 sometimes "over-generalized." For example, if you searched for "shredded chicken," GPT-4 might think a "Chicken Shredder" tool is highly relevant because the words match. But the specialized LLaMA2 understood that if you are searching for food, you probably don't want the tool to make it. It understood the intent, not just the words.

5. Why This Matters for You

Better Shopping: You get results that actually match what you are looking for, not just words that look similar.
Cost & Privacy: Because they used the "cheat sheet" method (LoRA) instead of rebuilding the whole brain, it is cheaper to run and keeps data more secure (since they don't have to send data to outside servers to use a giant model).
Business Success: When customers find what they want faster, they are happier and more likely to buy.

In a nutshell: The paper shows that by giving a smart, general AI a specialized "cheat sheet" for shopping, Walmart can understand what customers really want much better than before, leading to a smoother, more accurate shopping experience without breaking the bank.

Technical Summary: Enhancement of E-commerce Sponsored Search Relevancy with LLM

Problem Statement
Sponsored search is a critical revenue stream for e-commerce platforms, relying on the accurate matching of advertiser keywords to user search queries. However, ensuring ad pertinence is complicated by the vast, evolving keyword space, the ambiguity of user and advertiser intentions, and the diversity of topics and languages. Traditional algorithmic approaches, such as keyword matching and collaborative filtering, often fail to capture the nuanced language of consumer queries (e.g., implied preferences for durability or price range). This limitation leads to sub-optimal search results, negatively impacting user satisfaction, click-through rates, and conversion rates. The challenge lies in developing a system that can interpret brief, complex query intents quickly, accurately, and cost-effectively within the constraints of a live e-commerce environment.

Methodology
The authors propose an advanced Ad Relevance Model built upon the LLaMA2 7B decoder-based large language model (LLM), specifically adapted for the e-commerce sponsored search context. The core technical innovation is the use of Low-Rank Adaptation (LoRA) to fine-tune the pre-trained model.

Architecture: The system utilizes a parameter-efficient fine-tuning (Peft) approach. Instead of retraining the entire 7B parameter model, LoRA introduces low-rank matrices ( $A$ and $B$ ) to the weight matrices of specific components, including the attention mechanism (query, key, value, and output projections) and feed-forward networks (gate, up, and down projections). The original weights ( $W$ ) remain frozen, while only the LoRA parameters are updated during training. This significantly reduces computational resources and training time.
Task Formulation: The model functions as a <query, ad title> classifier. It takes a user query and an ad item title as input and predicts a relevance score, categorizing the pair into one of three classes: Relevant, Partially Relevant, or Irrelevant.
Training Data: The model was trained on a dataset derived from Walmart's query logs, comprising 250,000 query-item pairs for training, with 56,000 pairs each for validation and testing. The data was annotated by third-party human evaluators on a three-point scale (0: Irrelevant, 1: Partially Relevant, 2: Relevant), with the majority vote determining the ground truth.
Optimization: The model was trained using the Adam optimizer with a learning rate of 1e-5 and a categorical cross-entropy loss function over 7 epochs on a setup featuring 4 Tesla V100 GPUs.

Key Contributions

LLM Adaptation for E-commerce: The paper demonstrates the successful adaptation of the open-source LLaMA2 7B model to understand the specific intents behind e-commerce queries (e.g., "energy-efficient LED bulbs") without requiring a separate, complex query understanding system.
Efficient Fine-Tuning via LoRA: The authors showcase a novel application of LoRA to fine-tune a large language model for sponsored search, achieving high performance while maintaining computational efficiency and operational privacy.
Novel Classification System: The development of a specialized <query, ad title> classifier that discerns relevance across three distinct categories, improving upon binary relevance models.
Automated Labeling Framework: The system facilitates continuous model training through an advanced auto-labeling capability, reducing reliance on manual data annotation.
Comprehensive Benchmarking: A rigorous comparative analysis against traditional NLP models (BERT-based Bi-Encoder and Cross-Encoder) and state-of-the-art models (GPT-4 using few-shot learning).

Experimental Results
The fine-tuned LLaMA2 7B model demonstrated superior performance across multiple metrics:

Accuracy: The model achieved 89.43% accuracy on the test dataset, outperforming the BERT-based Cross-Encoder (86.27%), the BERT-based Bi-Encoder (74.42%), and GPT-4 (63.02%).
Precision and Recall: The model maintained a balanced precision of 89.75% and recall of 89.22%, resulting in an F1-score of 89.41%. This balance is critical for minimizing both false positives and false negatives in ad placement.
Relevance Ranking: In terms of Normalized Discounted Cumulative Gain (NDCG), the model achieved 0.7142 at NDCG@4 and 0.6774 at NDCG@8, outperforming both BERT variants.
Qualitative Analysis: Case studies indicated that while GPT-4 tended to overgeneralize (e.g., classifying a "Chicken Shredder" as highly relevant to "shredded chicken"), the fine-tuned LLaMA2 model exhibited better discernment of indirect relevance and context, correctly identifying items like "McKesson Medical Tape" for "first aid tape" while avoiding false positives on unrelated items like "glass cleaner" for "ZEISS lens wipes."

Significance and Claims
The paper claims that integrating LoRA with the LLaMA2 7B model represents a significant stride in customizing language models for e-commerce applications. The significance of this work lies in its ability to deliver a triad of benefits essential for modern digital marketplaces:

Enhanced Search Accuracy: By better understanding complex query intents, the model improves the relevance of sponsored results, directly boosting user satisfaction and business metrics.
Cost Efficiency: The LoRA approach allows for efficient fine-tuning without the extensive computational resources required for full model retraining, making it scalable for large-scale e-commerce platforms.
Operational Privacy: Unlike using external APIs (such as GPT-4), the use of an open-source, self-hosted model ensures data privacy and security, a critical factor for enterprise e-commerce environments.

The authors conclude that while the evaluation focused on sponsored search, the methodology is versatile and applicable to broader e-commerce search contexts, offering a practical and impactful solution for improving online retail experiences.