Technical Summary: Fortress – A Case Study in Stabilizing Search Recommendations

Problem Statement

In search and recommendation systems, predictive models frequently suffer from temporal instability. This occurs when specific input features cause output scores for the same query-entity pair to fluctuate significantly across different time periods. Such volatility degrades model reliability and user experience, particularly in multi-stage systems where consistent predictions are essential for downstream decision-making.

While recent advancements in Large Language Models (LLMs) and BERT-based semantic features have improved contextual relevance and generalization, they often lack full coverage of queries or entities. Conversely, engagement-based features (derived from organic user behaviors like clicks and searches) offer strong predictive power but are prone to introducing temporal inconsistencies. The core challenge addressed by this work is the trade-off between the high predictive accuracy of engagement signals and the score volatility they introduce.

Methodology: The Fortress Framework

The authors propose Fortress, a general framework designed to enhance model stability and accuracy by identifying and pruning features that contribute to inconsistent prediction scores over time. The framework operates on the insight that while certain features may be predictive in isolated snapshots, they can introduce volatility when evaluated across time.

Fortress follows a four-step process:

1. Data Collection: The system collects historical snapshots, which are temporally partitioned datasets capturing score fluctuations for the same entity across different periods.
2. Unstable Sample Identification: The method identifies samples exhibiting high score variance across these snapshots. Specifically, it targets the 75th percentile of samples with the highest Coefficient of Variation (CV).
3. Feature Isolation and Pruning: For the identified high-variance samples, the CV is computed for each individual feature. Features with the highest CVs are flagged as candidates for removal. The framework employs a greedy search approach: it iteratively removes candidate features and retrains the model. Crucially, feature removal only proceeds if it results in a statistically significant improvement in PR-AUC on the validation set compared to the baseline.
4. Model Retraining: The final model is retrained using the reduced feature set (excluding the instability-inducing features) and all available historical snapshots.

This approach fits within the class of "wrapper" feature selection methods, where feature subsets are chosen based on validation performance.

Experimental Setup

The framework was validated on a large-scale query-to-app relevance model within a commercial app marketplace. The task involved classifying query-app pairs as relevant (GOOD/EXCELLENT) or irrelevant (BAD).

• Data: The dataset consisted of daily snapshots, split into 70% training, 15% validation, and 15% test sets. Approximately 40% of pairs were labeled BAD.
• Features: The model utilized two primary categories:
  • Semantic Representation (SR): LLM-based features (using a proprietary Foundation Model) and DistilBERT-based features (two-tower architecture with triplet loss). These provide content-based relevance independent of engagement.
  • Engagement Features: Non-personalized, highly aggregated historical user interactions (clicks, searches). These were refreshed several times daily and constituted the primary source of score instability.
• Model: An XGBoost classifier was used for its speed and robustness to noise.
• Metrics:
  • PR-AUC: To assess classification performance.
  • Coefficient of Variation (CV): Defined as the standard deviation divided by the mean of prediction scores across snapshots, used to measure score stability.

Key Results

The experiments demonstrated that while adding engagement features improves classification performance, it simultaneously increases score volatility. Fortress successfully mitigated this trade-off:

• Stability Improvement: Compared to the baseline using all features, Fortress reduced the Coefficient of Variation (CV) by 3.44% (from 0.5462 to 0.5274).
• Accuracy Improvement: The framework achieved a 0.24% increase in PR-AUC (from 0.9572 to 0.9595), surpassing both the "All features" baseline and the "Multi-snapshot with all features" approach.
• System-Level Impact: As a downstream effect of improved stability, Fortress reduced the "flip-flop" rate (pairs appearing in the final recommendation stage in one snapshot but absent in another) by 25.8% globally. Regional reductions ranged from 14.7% to 49.6%.
• Feature Analysis: The study confirmed that some features beneficial in a single snapshot degrade performance when evaluated across time, reinforcing the necessity of temporal multi-snapshot analysis for robust feature selection.

Significance and Claims

The paper positions Fortress as a principled method for incorporating engagement signals while managing their inherent volatility. The authors claim that:

1. General Applicability: While demonstrated on a query-to-app relevance task, the technique is broadly applicable to any scenario where a scoring model evaluates the same entity over time, such as user-item recommendations.
2. System Reliability: In multi-stage retrieval and ranking pipelines, volatility in production scores can propagate and cause significant, undesirable shifts in system dynamics. Fortress addresses this by maintaining scoring stability, which is critical for ensuring reliability and performance.
3. Balanced Optimization: The framework successfully resolves the tension between the generalization of semantic features and the predictive power of engagement features, leading to models that are both more stable and more accurate.

The work concludes that delivering a high-quality user experience requires recommendation systems to produce results that are not only relevant but also consistent and predictable over time.