One Hierarchy, Two Systems: Semantic Product IDs for Discovery-Surface Ranking and Search-Page Query Reformulation
This paper demonstrates that a single, learned hierarchical Semantic ID representation can effectively unify product discovery and search tasks by enabling improved personalized ranking through aggregated behavioral signals and more effective query reformulation via concept-grounded navigation, all while preserving task-specific contexts.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast digital marketplaces where we buy groceries, clothing, and household goods, a single product often wears many different faces. A specific brand of milk might be listed by one store under one name and by another store under a completely different code, even though the bottle on the shelf is identical. This fragmentation makes it difficult for computers to understand that a customer who buys one version of a product is likely interested in the other. To solve this, researchers have long tried to organize products into categories, much like a library uses a filing system. However, these human-made categories are often too broad to capture the subtle differences between items, while the raw data from each store is too messy to connect the dots. The challenge lies in finding a way to teach a computer to see the relationships between products without losing the specific details that matter to a shopper.
A team of researchers at DoorDash set out to solve this problem by creating a new way to represent products, one that acts as a shared language for both recommending items and helping people search for them. They developed a system that converts product information into a short, unique code, similar to a digital fingerprint. When they analyzed these codes, they discovered they naturally formed a hierarchy, like a set of nested boxes. The outer layers of the code group products into broad categories, while the inner layers drill down into increasingly specific details. This structure allowed the system to understand that two products are related because they share the beginning of their codes, even if the stores listing them use different names. The researchers then tested whether this single hierarchy could power two very different tasks: deciding which products to show a customer on a browsing page, and suggesting better search terms when a customer types a query.
The researchers first applied this hierarchy to the problem of ranking products on discovery pages, where users browse through carousels of items like "Summer Grilling" or "Beauty." In the past, the system struggled to learn a customer's preferences because it treated every store's listing as a separate entity. If a user bought a specific type of coffee at one store, the system did not automatically know they might like the same coffee at a different store. By using the new hierarchical codes, the system could group these similar items together and aggregate the customer's history across all of them. This allowed the ranking algorithm to recognize a preference for a product concept rather than just a single store's listing. When they tested this in the real world, the system became significantly better at predicting what a customer would want to add to their cart. The results showed a noticeable increase in engagement, with customers adding more items to their carts, particularly in the top positions of the browsing lists. Perhaps more importantly, the system began to show a wider variety of products, giving less popular items a fair chance to be seen rather than just repeating the same best-sellers.
The same hierarchical structure was then used to improve the search experience, specifically the suggestions that appear when a user types a query. Traditionally, search systems suggest new terms by looking at what people typed next in their search history. This approach often fails because it gets confused by spelling mistakes, abbreviations, or the fact that the same words can mean different things in different contexts. The new system instead translates the user's search into the product concepts defined by the hierarchical codes. It then looks for transitions between these concepts rather than between raw words. For example, if a user searches for "milk," the system understands this as a product concept and can suggest moving laterally to "cereal" or refining the search to "whole milk," depending on what the user actually clicked on in the past. This method proved to be more precise than previous approaches. In tests, the new system generated suggestions that were more distinct and useful, helping users find what they were looking for faster. When deployed to real users, the system reduced the number of scrolls and clicks needed to find a product, meaning shoppers reached their desired items with less effort.
The study demonstrates that a single, learned hierarchy of product codes can serve as a powerful foundation for both recommending items and refining search queries. By moving away from rigid, human-made categories and instead using a structure that the computer learned directly from the data, the researchers created a system that is both flexible and precise. The findings suggest that when computers understand the relationships between products at multiple levels of detail, they can better serve the specific needs of individual shoppers. The results were not just theoretical; they were measured in real-world experiments where the new system consistently outperformed the old methods, leading to more satisfied customers and a more efficient marketplace. This work highlights how a shared understanding of product structure can bridge the gap between the chaotic reality of millions of different listings and the personalized experience every shopper expects.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.