Technical Summary: A Statistical Framework for Learning Preferences from the Past

Problem Statement

In various real-world domains such as online recommendation, consumer choice modeling, and streaming services, individuals make repeated selections from a fixed set of options. Accurately estimating underlying user preferences is critical for generating personalized future recommendations. While existing methods often rely on frequency of choice, this paper argues that the intensity of a choice (e.g., spending amount, viewing duration, or order value) provides a stronger quantification of preference than frequency alone.

The core challenge addressed is the estimation of user preference probabilities based on past decisions under a natural monotonicity assumption: options chosen more frequently or more intensely in the past are more likely to be chosen again in the future. The authors aim to move beyond heuristic curve-fitting and parametric models (such as those used for ant colony foraging) to develop a statistically rigorous, non-parametric framework that accounts for both the history of choices and their varying intensities.

Methodology

Model Formulation

The authors propose a probabilistic framework where a user's choice at time $t+1$ is conditioned on the relative intensity of past choices up to time $t$.

• Data Structure: For $N$ users, each makes a sequence of choices from two options ($c_1$ and $c_0$) over a discrete time window. Each choice is associated with an intensity $W_t$.
• Relative Intensity ($R_t$): Defined as the weighted proportion of times option $c_1$ was chosen relative to the total intensity of all choices up to time $t$:
  $$R_t = \frac{\sum_{i=1}^t U_i W_i}{\sum_{i=1}^t W_i}$$
  where $U_i \in \{0, 1\}$ indicates the choice and $W_i$ is the intensity.
• Preference Function ($h$): The probability of choosing $c_1$ at time $t+1$ is modeled as $P(U_{t+1}=1 | R_t) = h(R_t)$.
• Constraint: The function $h: [0, 1] \to [0, 1]$ is assumed to be non-decreasing, capturing the reinforcement dynamic where higher past relative intensity increases the probability of future selection.

Connection to Stochastic Processes

The framework draws inspiration from the generalized elephant random walk (introduced by [16]). In the special case where choice intensities are equal ($W_t \equiv 1$), the sequence of choices in this model corresponds one-to-one with the steps of a generalized elephant random walk. This connection motivates the use of non-parametric methods rather than the parametric models previously applied to ant path selection (e.g., [7], [9]).

Estimation Procedure

The authors employ Non-Parametric Maximum Likelihood Estimation (NPMLE) under the monotonicity constraint.

1. Likelihood Construction: The log-likelihood is formulated based on the observations of choices and their corresponding relative intensities.
2. Optimization: The estimator $\hat{h}_N$ is defined as the unique non-decreasing, right-continuous step function that maximizes the log-likelihood. This is solved as a convex optimization problem (isotonic regression) using the Pool Adjacent Violators Algorithm (PAVA).
3. Confidence Sets: Instead of relying on Wald-type intervals (which require estimating nuisance parameters like derivatives of $h$ and the distribution of $R_t$), the authors construct confidence sets by inverting likelihood ratio tests. This approach leverages the asymptotic distribution of the likelihood ratio statistic, which is pivotal and does not require nuisance parameter estimation.

Key Theoretical Results

The paper establishes rigorous asymptotic properties for the proposed estimator under standard regularity assumptions (bounded support of choice counts, smoothness of the underlying distribution, and non-zero derivative of $h$).

1. Asymptotic Distribution of the Estimator:
   The estimator $\hat{h}_N(r_0)$ at a fixed point $r_0$ converges at a rate of $N^{1/3}$ (slower than the standard $N^{1/2}$ rate typical in parametric settings). The limiting distribution is characterized by the Chernoff distribution (the distribution of the location of the minimum of a two-sided Brownian motion with quadratic drift).
   $$N^{1/3}(\hat{h}_N(r_0) - h(r_0)) \xrightarrow{d} C \cdot \arg\max_{x \in \mathbb{R}} (B(x) - x^2)$$
   where $B(x)$ is a two-sided Brownian motion.

2. Likelihood Ratio Statistic:
   The likelihood ratio statistic for testing $H_0: h(r_0) = h_0$ converges in distribution to a random variable $D$, defined as the integral of the squared difference between the greatest convex minorant of a Brownian motion with drift and its restriction.
   $$2 \log \lambda_N(h_0) \xrightarrow{d} \int \left( (g_{1,1}(x))^2 - (g^0_{1,1}(x))^2 \right) dx$$
   This result allows for the construction of confidence intervals without estimating the derivative $h'(r_0)$ or the density of the covariates, avoiding the "nuisance parameter" problem common in monotone function estimation.

Empirical Results

Simulation Studies

The authors conducted extensive simulations across six configurations varying the number of choices (constant vs. random) and intensity regimes (equal, uniform, and persistent).

• Performance Metrics: They evaluated Test Error (predictive accuracy against an oracle) and Expected Calibration Error (ECE).
• Findings:
  • Both test error and ECE decreased steadily as the sample size $N$ increased, demonstrating consistency.
  • The estimators were well-calibrated even at moderate sample sizes.
  • The 95% confidence sets exhibited desirable behavior: their length decreased monotonically with $N$, and coverage probabilities remained close to the nominal 0.95 level across all configurations.

Real-World Application

The framework was applied to the MovieLens (20M) dataset to estimate user preferences for movie genres (e.g., Action vs. Romance).

• Setup: Users' ratings were treated as choice intensities. The model was trained on choices 11–15 and tested on choices 16–20.
• Intensity Comparison: The authors compared a "Constant Intensity" model (all choices weighted equally) against a "Rating-based Intensity" model (weights equal to the user's rating).
• Results: The rating-based intensity did not yield substantial improvements over the constant intensity model in terms of predictive accuracy or calibration. This suggests that for this specific dataset, the simpler constant-intensity model captures the essential structure of the data.
• Interpretability: The estimated preference functions and their confidence sets provided actionable insights. For instance, the confidence intervals allowed for risk-aware decision-making, distinguishing between users with high preference certainty and those with high uncertainty.

Significance and Claims

The paper claims to provide a novel statistical framework that bridges the gap between reinforcement dynamics observed in nature (ant colonies) and human decision-making. Its primary contributions are:

1. Non-Parametric Flexibility: Unlike previous parametric approaches, this framework does not impose strong functional form assumptions, allowing it to capture diverse user behaviors.
2. Incorporation of Intensity: It explicitly models the intensity of choices, acknowledging that not all user actions carry equal weight.
3. Statistical Rigor: It provides the first theoretical guarantees (consistency, asymptotic distribution, and confidence sets) for preference estimation under monotonicity constraints in this specific reinforcement setting.
4. Practical Utility: The likelihood-ratio-based confidence sets offer a principled way to quantify uncertainty without complex nuisance parameter estimation, making the method robust and applicable to real-world recommendation systems.

The authors conclude that while the framework is currently designed for stationary settings, it serves as a foundational step for future work involving non-stationary dynamics and the inclusion of user-specific covariates.