Technical Summary: JourneyFormer

Problem Statement

In two-sided e-commerce platforms like Airbnb, search ranking is critical for connecting guests with suitable stays. While sequence modeling has gained traction in recommendation systems for capturing user intent, its practical deployment in production search ranking faces significant hurdles. Specifically, Airbnb guest journeys are characterized by:

1. Long and Exploratory Sequences: Guests often browse extensively across different destinations and dates, resulting in sequences spanning hundreds of thousands of events. These sequences contain significant noise, as many views are exploratory rather than indicative of booking intent.
2. Event Complexity: Guest interactions vary widely in frequency and meaning. High-frequency events (e.g., views) often lack clear intent, while low-frequency events (e.g., bookings, cancellations, reviews) are highly informative but sparse.
3. Sparse Labels: Unlike social media platforms that optimize for engagement, Airbnb prioritizes booking conversion. The target signal (booking) is extremely sparse compared to engagement signals, making it difficult to train models that generalize well without overfitting to noisy engagement data.
4. Production Scalability: Previous approaches relied on hand-crafted features aggregated over time windows. While effective, these approaches became difficult to maintain and scale as the number of features grew, often failing to capture complex, non-linear guest intentions.

Methodology: JourneyFormer

JourneyFormer is a sequence modeling solution designed to encode guest journeys into compact representations for search ranking. The architecture and design choices are driven by production constraints and the specific nature of Airbnb data.

1. Data Selection and Sequence Construction

To manage the scale and noise of guest data, the authors employ a frequency-based event selection strategy:

• Long-term Sequence: Captures infrequent, high-value events (bookings, reviews, cancellations, CS tickets) from the past 7 years.
• Short-term Sequence: Captures high-frequency events (views) from the past 21 days.
• Capping: Sequences are capped at 80 events for long-term and 200 for short-term to ensure GPU memory efficiency during training, covering the top 2% of guests with the longest histories.

2. Feature Representation

A shared feature pool is used across guest sequences, search queries, and listing candidates to facilitate embedding learning:

• ID Features: Listing IDs, Host IDs, and hierarchical S2Cell IDs (geographical indexing from level 0 to 13) are processed using Unified Embedding. This technique uses a single shared embedding table with random hashing seeds to handle high-cardinality IDs efficiently, outperforming separate ID embeddings.
• Categorical & Continuous Features: Categorical features are embedded, while continuous features (e.g., price) are normalized via batch normalization.
• Raw Semantics: The model uses raw features without extensive manual feature engineering, delegating interaction modeling to the neural architecture.

3. Model Architecture

• Sequence Encoder: A Transformer-based encoder with multi-head self-attention and causal masking is used. The causal mask ensures that future events do not influence the encoding of past events, preventing information leakage. Notably, the authors found that feed-forward layers (common in standard Transformers) degraded performance and slowed training, so they were excluded.
• Ranking Model: The sequence embedding is concatenated with query and listing candidate embeddings. A lightweight Multi-Layer Perceptron (MLP) scores the listings.
• Label Attribution: To address label sparsity, a booking is attributed to all searches conducted in the preceding 7 days. Only searches with at least one booking are used for training, with up to four searches sampled per day per guest to maintain diversity.

4. Training and Serving Strategies

Several strategies were implemented to accelerate training and inference by approximately 4x:

• Sequence Bucketization: Sequences are grouped by length (e.g., 4, 16, 44, 88, 148, 280) to minimize padding overhead during batching.
• Batching of Searches: Intermediate embeddings from the sequence encoder (for events preceding the final one) are reused as sequence embeddings for subsequent searches within the same journey, reducing redundant forward/backward passes.
• Sparse Calculation: Only non-padded searches are passed to the ranking model, avoiding unnecessary computation on padded data.
• Hybrid Serving: The sequence encoder is run in batch mode (daily) to precompute guest embeddings, while the ranking model runs in real-time using these precomputed embeddings, balancing latency and cost.

Key Contributions

1. Production-Oriented Design: The paper details the specific trade-offs required to deploy sequence models in an industrial setting, moving beyond pure algorithmic novelty to address data selection, label attribution, and serving scalability.
2. JourneyFormer Deployment: The successful deployment of a Transformer-based sequence model in Airbnb's search ranking, replacing hand-crafted feature engineering with learned sequence embeddings.
3. Empirical Validation: Extensive offline and online analysis demonstrating that the model improves both ranking metrics and key business outcomes.
4. Lessons Learned: The authors provide insights into design choices that were explored but rejected (e.g., bidirectional encoders, real-time sequence inference, multi-task learning with view labels) due to lack of online gain or increased complexity.

Results

Offline Performance

• NDCG Improvement: JourneyFormer significantly outperformed the strong production baseline (which relied on hand-crafted features). The full model (using both long and short-term sequences) achieved a +1.48% improvement in NDCG.
• Sequence Length Analysis: Performance improved with longer long-term sequences. However, short-term sequences showed peak performance with 11-20 events, declining thereafter, validating the hypothesis that excessive views introduce noise.
• Design Ablations: Removing ID embeddings (especially S2Cell IDs) or adding feed-forward layers resulted in performance drops, confirming the importance of location granularity and the specific architecture choices.

Online A/B Testing

• Search Ranking: Over a three-week period, JourneyFormer-Full showed significant gains in key business metrics:
  • Bookers: +0.55%
  • Booked Nights: +0.82%
  • Views: +0.90%
• Promotional Emails: When applied to email listing ranking, the model increased email click-through rates by +5.04%, along with significant gains in bookers and booked nights.
• Distribution Shift: The model maintained consistent performance (>1.4% NDCG improvement) over a month of evaluation data post-training, indicating robustness to distribution shifts and reducing the need for frequent retraining.

Significance and Claims

The paper claims that JourneyFormer successfully bridges the gap between theoretical sequence modeling and industrial-scale deployment. Its significance lies in:

1. Proving Effectiveness in Sparse Label Environments: Demonstrating that sequence modeling can effectively infer booking intent despite the sparsity of conversion labels and the noise of exploratory browsing.
2. Scalability: Showing that with careful engineering (bucketization, batched search inference, unified embeddings), complex sequence models can be trained and served efficiently at the scale of Airbnb's millions of daily events.
3. Business Impact: Providing concrete evidence that sequence modeling leads to tangible business growth (more bookings and nights) in a highly optimized, mature search ranking system where even small improvements are difficult to achieve.
4. Practical Guidance: Offering a roadmap for other practitioners on what works (causal masking, hierarchical location IDs, frequency-based capping) and what does not (real-time sequence inference for retargeting, multi-task learning with noisy engagement labels) in e-commerce search ranking.