Technical Summary: BACH (Bayesian Admixture of Contrastive Heads)

1. Problem Statement

Two-tower retrievers are the standard for large-scale candidate generation in recommendation systems, mapping users and items into a shared embedding space scored by inner product. However, standard single-vector models struggle to represent users with diverse, weakly related interests, often forcing all interests to collapse into a single point in the embedding space. This leads to retrieval slates that over-represent dominant interests while under-serving others.

Multi-interest models attempt to solve this by emitting $k$ user embeddings (heads) and scoring items by the maximum inner product across these heads. However, existing approaches suffer from two critical limitations:

1. Routing Collapse: Training via "hard-routing" (where only the head with the highest score to the positive item receives gradients) creates a winner-take-all dynamic. Heads initialized slightly worse are selected less frequently, receive fewer updates, and progressively collapse into copies of the dominant head, leaving the effective capacity below $k$.
2. Lack of Per-User Interest Weighting: Existing methods treat all $k$ heads symmetrically, providing no per-user estimate of how likely each head is to explain the next item. Consequently, serving rules either pick the maximum-scoring head or take a union of top lists under an implicit uniform prior, failing to reflect the heterogeneous importance of interests for specific users.

2. Methodology: BACH

The authors propose BACH (Bayesian Admixture of Contrastive Heads), which reframes multi-interest two-tower retrieval as a per-user mixture of softmaxes, fitted via variational inference.

Generative Model

BACH treats the $k$ user heads as components of a finite mixture model. For a user $u$ and item $i$:

• A latent component indicator $z_n \in \{1, \dots, k\}$ is introduced with a per-user prior $\pi_u$ (a mixture weight vector).
• Conditional on $z_n = r$, the item is generated from a sampled-softmax distribution defined by the $r$-th user embedding $u^{(r)}$.
• The observed data log-likelihood is the marginal over the latent heads: $p(i|u) = \sum_r \pi_u^{(r)} p_r(i|u)$.

Variational Inference and Recognition Posterior

Direct optimization of the log-likelihood is challenging due to the log-of-sum term. BACH employs variational inference to maximize the Evidence Lower Bound (ELBO).

• Recognition Posterior: Instead of using the model's own negative-dependent posterior (which is intractable at serving time), BACH introduces a spherical recognition posterior $\gamma_n$ over the heads.
• Kernel Choice: The posterior is modeled using a spherical density (defaulting to Power-Spherical, with von Mises-Fisher as an alternative). The density is parameterized by the head direction $u^{(r)}$ and a concentration parameter $\kappa_r$.
• Responsibilities: The variational distribution (responsibility) $\gamma_n^{(r)}$ is computed as:
  $$\gamma_n^{(r)} = \frac{\pi_u^{(r)} \sigma_r(n)}{\sum_{r'} \pi_u^{(r')} \sigma_{r'}(n)}$$
  where $\sigma_r(n)$ is the spherical density value based on the alignment between the head and the positive item.

Optimization

The model maximizes the ELBO jointly over:

1. Embeddings ($\theta, \phi$): The user heads and item tower are trained via a responsibility-weighted contrastive loss. Unlike hard-routing, gradients flow through all $k$ heads, weighted by their responsibility $\gamma_n^{(r)}$, mitigating routing collapse.
2. Mixture Weights ($\pi_u$): A gating tower $\pi_\xi(x_u)$ produces per-user weights. These weights act as a prior in the ELBO and are reused at serving time to weight the per-head retrieval.
3. Concentrations ($\psi$): The concentration parameters $\kappa_r$ are learned to fit the sharpness of the contrastive posterior. Crucially, because $\kappa_r$ enters the ELBO only through the KL-divergence term, they are self-regularized and require no explicit prior or clipping.

Serving

At serving time, BACH ranks items by the self-normalized mixture density:
$$m(i|u) = \sum_{r=1}^k \pi_u^{(r)} C_d(\kappa_r) \omega_{\kappa_r}(\langle u^{(r)}, v_i \rangle)$$
This allows for efficient retrieval:

• Soft Ranking: Items are ranked by a soft mixture rather than a hard max, allowing collapsed or low-mass heads to contribute minimally.
• Global Codebook Variant: BACH supports a "global" mode where heads are shared across all users (a fixed codebook of topics), while $\pi_u$ remains personalized. This enables precomputable retrieval, where per-head candidate lists are computed offline and reweighted by $\pi_u$ at serving, eliminating per-user nearest-neighbor queries.

3. Key Contributions

• Variational Formulation: Casting multi-interest retrieval as a per-user mixture of softmaxes fitted by variational inference, replacing hard arg-max routing with soft responsibilities.
• Mitigation of Routing Collapse: By distributing gradients across all heads based on soft responsibilities, BACH prevents the "winner-take-all" dynamic common in hard-routing multi-interest models.
• Per-User Interest Weighting: The model learns explicit per-user mixture weights ($\pi_u$) that quantify head importance, which are reused at serving to weight retrieval results, addressing the lack of interest heterogeneity in existing methods.
• Self-Regularized Concentrations: The concentration parameters $\kappa_r$ are fit to the data via the ELBO's KL term, requiring no external priors and remaining bounded naturally.
• Serving-Consistent Scoring: The paper demonstrates that training with a "serving-consistent" objective (scoring every candidate by its best head, rather than just the positive) outperforms traditional target-routed training.

4. Experimental Results

The authors evaluated BACH on three large-scale benchmarks: MovieLens-20M, Taobao, and Netflix.

• Performance: BACH outperformed all baselines (single-vector, hard-routing multi-interest like MIND/ComiRec, and max-routing variants) across all metrics (Recall@K, NDCG@K, AUPRC) and at every head count ($k \in \{8, 16, 32\}$).
• Top-of-Ranking Gains: Improvements were most significant at the top of the ranking (AUPRC, R@10), indicating better alignment with user intent.
• Routing Ablation: The paper found that "all-multihead" (scoring every candidate by its best head) significantly outperformed "pos-multihead" (target routing), with gaps widening as $k$ increased. BACH further improved upon the all-multihead baseline.
• Kernel Variants: The Power-Spherical (p-BACH) and von Mises-Fisher (v-BACH) variants performed comparably, with gains attributed to the variational mixture structure rather than the specific spherical kernel.
• Global Codebook: The global-codebook variant (shared heads, personalized weights) achieved strong performance (beating popularity baselines by 2–5x) while offering precomputable, low-latency serving and cold-start robustness, though it traded some accuracy compared to fully personalized heads.

5. Significance and Claims

The paper claims that BACH provides a principled probabilistic framework for multi-interest retrieval that directly addresses the structural flaws of hard-routing methods. By treating the problem as a variational mixture, it achieves:

1. Robustness: Elimination of head collapse without complex regularization heuristics.
2. Calibration: Production of calibrated per-user interest weights that improve serving consistency.
3. Flexibility: A spectrum of deployment options ranging from fully personalized heads to a precomputable global codebook, allowing trade-offs between accuracy and serving efficiency.

The authors emphasize that the method improves retrieval quality on real-world, large-scale datasets while maintaining a tractable serving architecture, making it a practical advancement over existing multi-interest baselines.