💬 NLP

A Modular LLM Framework for Explainable Price Outlier Detection

This paper proposes a modular, agentic Large Language Model framework that detects and explains product price outliers by classifying relevant similar items, assessing their relative utility across key attributes, and reasoning through these comparisons to achieve over 75% agreement with human auditors while outperforming existing zero-shot and retrieval-based methods.

Original authors: Shadi Sartipi, John Wu, Sina Ghotbi, Nikhita Vedula, Shervin Malmasi

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shadi Sartipi, John Wu, Sina Ghotbi, Nikhita Vedula, Shervin Malmasi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a massive, bustling digital supermarket (like Amazon). You pick up a jar of peanut butter priced at $50. Your brain immediately screams, "That's wrong! A jar of peanut butter shouldn't cost that much!"

In the past, computers tried to catch these pricing errors using rigid math rules, like saying, "If the price is more than 3 times the average, flag it." But this is like a security guard who only looks at height. If a giant peanut butter jar is $50, the guard might let it through because it's "big," even though it's still a rip-off. Or, if a tiny, fancy jar is $10, the guard might flag it as an error, even though it's actually a fair price for a luxury item.

This paper introduces a new, smarter system: The "AI Detective" Framework.

Instead of just doing math, this system acts like a human auditor who uses common sense and reasoning. Here is how it works, broken down into three simple steps using a creative analogy:

The Analogy: The "Smart Shopper" Detective

Imagine you are the detective trying to figure out if a specific product (the Target) is overpriced. You don't just look at the price tag; you go through a three-step investigation.

Step 1: The "Who's in the Room?" Agent (Relevance)

First, the AI asks: "Who else is in the room that I should compare this to?"
It doesn't just grab random items. It looks for neighbors—products that are similar enough to be fair comparisons.

The Mistake of Old Systems: They might compare a $50 jar of generic peanut butter to a $50 jar of organic, gluten-free, almond-butter spread. That's a bad comparison!
The AI Detective: It filters out the noise. It says, "No, that almond butter is too fancy. Let's look at the standard peanut butter jars from the same brand." It builds a shortlist of fair competitors.

Step 2: The "Value Judge" Agent (Utility Assessment)

Now, the AI looks at the shortlist and asks: "Are these competitors actually the same, or is one of them 'better'?"
This is where the AI uses common sense. It compares specific features like:

Brand: Is one a famous name and the other a generic store brand?
Size: Is one a 16oz jar and the other a 32oz jar?
Features: Is one organic or non-GMO?

The AI gives each competitor a score:

Better (+1): This competitor is fancier or bigger.
Worse (-1): This competitor is smaller or cheaper.
Mixed (0): They are about the same.

The Magic: If the Target product is $50, but the AI finds a competitor that is Bigger and Better but only costs $20, the AI realizes, "Wait a minute. If the better one is cheap, why is our Target so expensive?"

Step 3: The "Final Verdict" Agent (Decision)

Finally, the AI gathers all the evidence and makes a decision. It uses a special map (a 4-quadrant chart) to visualize the data:

The "Anomaly Zone": If there are products that are Better than your target but Cheaper, your target is likely a rip-off.
The "Safe Zone": If the cheaper products are actually Worse (smaller, generic), then your high price might be justified.

The AI then gives a reasoned explanation, not just a "Yes/No." It says: "I flagged this as an error because I found a 32oz jar of the same brand for $25, while this 16oz jar is $50. That's a bad deal."

Why is this a big deal?

It Explains Its Work: Old systems are "black boxes." They say "Error!" and you have no idea why. This system acts like a transparent teacher, showing its homework: "I compared it to X, Y, and Z, and here is why X is cheaper." This helps human auditors trust the system.
It's Flexible: The system can be tuned. If you want to be super strict (catch every possible error), you can tell it to be aggressive. If you want to avoid annoying customers with false alarms, you can tell it to be more careful.
It Works Like Humans: The paper tested this against human experts. The AI agreed with the humans 76% of the time, which is a huge success for a computer. It even caught tricky cases that simple math missed.

The Bottom Line

This paper proposes a way to teach computers to shop like humans. Instead of just crunching numbers, the AI learns to understand context, quality, and value. It's like upgrading from a calculator to a smart, reasoning assistant that can tell you not just that a price is wrong, but why it feels wrong.

This is a step toward Trustworthy AI in finance and retail, where machines don't just make decisions, but explain them in a way that makes sense to us.

1. Problem Statement

In large-scale e-commerce, price outlier detection (specifically identifying erroneously high prices) is critical for maintaining revenue, competitiveness, and consumer trust.

Limitations of Classical Methods: Traditional approaches (statistical thresholds, Isolation Forests, Z-scores) rely on heuristic rules or univariate/multivariate distributions. They function as "black boxes," offering no rationale for why a price is flagged, and fail to capture rich semantic relationships between products (e.g., how brand, pack size, or features influence value).
Limitations of Existing AI: Current Retrieval-Augmented Generation (RAG) or embedding-based systems struggle to capture price-relevant semantic nuances. Retrieved "similar" products often differ in key price-influencing attributes (e.g., a 10oz vs. 16oz pack), introducing noise.
The Gap: There is a lack of automated systems that emulate the human audit logic of: (1) finding truly comparable substitutes, (2) weighing value dimensions (brand, quantity, features), and (3) generating an interpretable justification for the decision.

2. Methodology: The Agentic LLM Framework

The authors propose a modular, three-stage Agentic LLM framework that treats price outlier detection as a structured reasoning task. The system processes a target product through three sequential agents:

Stage 1: Relevance Classification Agent

Goal: Identify "neighbor" products that are semantically relevant substitutes for the target.
Process:
1. Candidate Generation: Uses proprietary product embeddings (trained via contrastive learning on titles/images) to retrieve top- $k$ nearest neighbors via cosine similarity.
2. Filtering: The LLM evaluates each candidate against disjunctive logic questions: "Would customers compare prices between the target and this product?", "Is it similar?", and "Can it serve the same use case?"
3. Output: A filtered set of "relevant" neighbors.

Stage 2: Utility Assessment Agent

Goal: Evaluate the relative value of the target product against each relevant neighbor based on price-influencing attributes.
Attribute Selection Configurations: The paper evaluates four strategies for selecting attributes to compare:
1. Generic: Fixed attributes (Build Quality, Features, Brand, Quantity) for all categories.
2. Static: Category-specific attributes generated via zero-shot LLM.
3. Dynamic: Attributes generated per target-neighbor pair.
4. Weighted Dynamic (W-Dynamic): Attributes generated per pair with assigned importance weights.
Scoring: For each attribute, the agent rates the neighbor as Better (+1), Mixed (0), or Worse (-1) compared to the target. These scores are aggregated to determine relative utility.

Stage 3: Reasoning-Based Decision Agent

Goal: Aggregate evidence to make a final outlier judgment.
Mechanism: The system maps the target and neighbors onto a 4-Quadrant Framework (Relative Price vs. Relative Utility):
- Anomaly Price (AP) Zone (Bottom-Right): Neighbors with Higher Utility but Lower Price. This supports the hypothesis that the target is overpriced.
- NOT AP Zone (Top-Left): Neighbors with Lower Utility but Higher Price. This refutes the anomaly hypothesis.
- Padding: The system introduces "price padding" (a minimum % price difference) and "utility padding" to filter out noise and uncertainty.
Decision Strategies:
1. Worse-Pricier Veto: If any relevant product exists in the NOT AP zone (worse utility but higher price), the target is not an outlier. (Prioritizes Precision).
2. Quadrant Voting: If the count of products in the AP zone $\ge$ count in the NOT AP zone, the target is an outlier. (Prioritizes Recall).

3. Key Contributions

Novel Framework: A novel agentic architecture that decomposes price outlier detection into interpretable sub-tasks (Relevance $\to$ Utility $\to$ Decision), mirroring human audit workflows.
Explainability: Unlike black-box statistical models, this system generates verifiable reasoning chains anchored in structured product data, allowing human auditors to verify which neighbors were compared and why.
Comprehensive Evaluation: Extensive ablation studies on hyperparameters (neighbor count, price padding, attribute selection strategies) and decision logic.
Practical Applicability: Demonstrated success in a real-world enterprise setting (Amazon), reducing the need for expensive manual audits while maintaining high alignment with expert judgment.

4. Experimental Results

The framework was evaluated on a dataset comprising a Silver Set (40 human-annotated products), a One-Sided Set (185 difficult non-outlier cases), Edge Cases (12 unique products), and a large Unannotated Set (5,400 products).

Performance vs. Baselines:
- Agentic Framework: Achieved 76.3% agreement with human auditors on the challenging one-sided set, with an F1 score of 0.55 and Perfect Precision (1.00) on the Silver Set.
- Zero-Shot LLM: 68% agreement, F1 0.43.
- RAG (Retrieval only): 23% agreement, F1 0.42.
Key Findings from Ablation:
- Attribute Selection: The Generic configuration (fixed attributes) provided the best balance (F1 0.55, 76.3% agreement). Complex dynamic configurations (W-Dynamic) increased agreement (81.6%) but drastically reduced F1 scores (0.20), suggesting overloading the LLM with judgment tasks harms consistency.
- Decision Strategy: The Worse-Pricier Veto strategy yielded high precision (1.0) and strong human agreement (76.3%), aligning with how human auditors typically make decisions (low cognitive load, high confidence).
- Hyperparameters: 7 neighbors and 50% price padding were identified as the "sweet spot" for balancing F1 score, agreement, and outlier rate.
Operational Metrics: The system maintained a 7.8% outlier flagging rate on the unannotated set (well below the 10% business guardrail), ensuring it does not overwhelm human reviewers.

5. Significance and Impact

Trustworthy AI: The framework bridges the gap between high-accuracy AI and the need for transparency in financial/retail systems. By providing "why" a price is flagged, it enables efficient human-in-the-loop verification.
Scalability: It offers a cost-effective alternative to manual auditing for millions of SKUs, capable of adapting to different accuracy requirements (e.g., high-precision for consumer-facing alerts vs. high-recall for internal audits).
Generalizability: The modular reasoning approach is applicable to adjacent domains such as product matching, fraud detection, and supply chain anomaly detection.

Conclusion: The paper demonstrates that agentic LLMs, when structured to mimic human reasoning steps (relevance filtering, utility assessment, and evidence aggregation), can significantly outperform traditional statistical and retrieval-based methods in detecting price anomalies while providing the explainability required for enterprise adoption.