Technical Summary: UniNote

Problem Statement

Item-to-Item (I2I) retrieval is a critical component of modern content platforms like Xiaohongshu, where user-generated "Notes" consist of heterogeneous modalities (text, images, videos, and embedded OCR). Current retrieval paradigms face three fundamental limitations when handling these complex, multimodal items:

1. Structural Disconnect: Standard dual-tower models (e.g., CLIP) encode modalities independently to ensure efficiency. This late-interaction mechanism fails to capture the fine-grained alignment required between text and visual components within a composite note, leading to a loss of local detail.
2. Granularity Mismatch: While Multimodal Large Language Models (MLLMs) excel at global reasoning, they often struggle with the hierarchical structure required for "local-to-global" retrieval (e.g., retrieving a full note based on a single image fragment). They are typically optimized for broad semantic descriptions rather than precise sub-component alignment.
3. Inefficiency in Ranking: Industrial pipelines often rely on a disjointed "retrieve-then-rerank" strategy. This involves using heavy generative models for reranking, creating computational bottlenecks. Furthermore, models trained on user interaction data (clicks/likes) often optimize for user interest rather than objective content relevance, making them unsuitable for strict semantic retrieval tasks.

Methodology

To address these challenges, the authors propose UniNote, a unified embedding model designed specifically for industrial I2I retrieval. The framework unifies representation learning and relevance ranking within a single model architecture through a novel two-stage training paradigm.

1. Task Definition and Data Construction

The authors define a comprehensive I2I task suite covering ten distinct retrieval categories, including:

• Atomic Alignment: Cross-modal search (image $\leftrightarrow$ text).
• Subordinate Retrieval: Local-to-global retrieval (e.g., image $\to$ full note).
• Semantic Extraction: Global-to-local retrieval (e.g., note $\to$ specific image).
• OCR Perception: Text-in-image search and OCR-to-note retrieval.
• Content Relevance: Note-to-note ranking.

To support these tasks, a high-quality dataset is constructed using real Xiaohongshu notes. A Multi-granularity Hard Negative Mining strategy is employed to enhance discriminative power. This includes:

• Modal Replacement: Substituting images with their textual descriptions to prevent the model from relying on low-level pixel shortcuts.
• Counterfactual Hard Negatives: Constructing negative pairs that maintain maximum content overlap with the positive context but strictly preclude valid membership relationships (e.g., using a note minus one specific image as a negative for that image).
• Soft Supervision: Utilizing similarity scores from a reference encoder to provide soft labels for distribution alignment.

2. Two-Stage Training Paradigm

Stage 1: Contrastive Supervised Fine-Tuning (SFT)
The goal is to transform a generative MLLM into a robust embedding model.

• Architecture: The model uses the "last-token" hidden state as the final representation, compressing heterogeneous inputs into a unified vector space.
• Optimization: The model minimizes the Jensen-Shannon (JS) divergence between the embedding-based distribution and the MLLM-derived soft-label distribution.
• MRL Integration: Matryoshka Representation Learning (MRL) is integrated to support nested dimensions (64, 512, 1024, 4096), allowing for flexible deployment based on cost-efficiency trade-offs.

Stage 2: Relative Reranking via Reinforcement Learning (RL)
This stage bridges the gap between retrieval (embedding) and ranking (relevance) without a separate reranker.

• Framework: The model is trained using Group Relative Policy Optimization (GRPO).
• Data Construction: For a given note, visual and textual components are partitioned to create a candidate sequence with incrementally decreasing content overlap, mixed with noise sequences.
• Reward Function: A multi-dimensional reward function is designed to optimize list-wise relevance, comprising:
  • Irrelevant Note Penalty: Penalizes noise notes ranked above relevant ones.
  • Base Relevant Reward: Rewards the recall of relevant samples.
  • Absolute Position Reward: Minimizes the distance between predicted and ground-truth positions.
  • Relative Order Reward: Ensures global consistency in the ordering of relevant notes.
• Outcome: The model learns to distinguish subtle content nuances and produce relevance-aware rankings in a single pass, eliminating the need for computationally expensive generative rerankers.

Key Contributions

1. Comprehensive I2I Task Suite: The definition of ten distinct retrieval categories spanning cross-modal alignment, information compression, and fine-grained local-global bidirectional retrieval, supported by a robust hard-negative mining dataset.
2. Unified Retrieval-Ranking Framework: The proposal of UniNote, which integrates embedding and relevance ranking into a single model via a two-stage training paradigm (Contrastive SFT followed by Relevance-aware RL).
3. SOTA Performance: Extensive evaluations demonstrating state-of-the-art results across all ten I2I tasks, with ablation studies confirming the synergy between the two training stages.
4. Industrial Deployment Validation: Successful integration of MRL for flexible dimension selection and validation of the model's performance in large-scale online and offline production environments at Xiaohongshu.

Experimental Results

• Benchmark Comparison: UniNote was evaluated against SOTA baselines (RzenEmbedding and Qwen3VL-Embedding-8B) on a dataset of 66k notes and 500k+ items.
  • UniNote achieved the best performance in Subordinate Retrieval (e.g., I2Note: 93.7% R@1 vs. 41.2% for baselines) and Semantic Extraction, tasks where baselines struggled due to their inability to model global-local relationships.
  • In OCR Perception, UniNote significantly outperformed baselines in OCR-to-Note and OCR-to-Image tasks, though it showed slightly lower performance in Image-to-OCR compared to some baselines.
  • For Note-to-Note relevance, UniNote achieved comparable or slightly improved recall (R@10: 99.5%) with significantly higher precision (P@5: 96.0% vs. 92.6% for SFT-only).
• Ablation Studies:
  • Removing hard negative mining (random sampling) resulted in a significant performance drop, validating the necessity of global hard negative mining and soft supervision.
  • Excluding rule-based heuristics and counterfactual negatives reduced the model's ability to distinguish subtle semantic differences.
  • Removing the RL stage reduced Recall@5 and Precision@5 for the Note-to-Note task by 1.4% and 3.4%, respectively.
• MRL Performance: At 512 dimensions, performance approximated full-dimensional features. Even at 64 dimensions, the model retained ~70% of its capability in key tasks, demonstrating effective trade-offs between storage and accuracy.
• Online Deployment: In online A/B testing, UniNote achieved high recall retention (85.6%–93.6%) across Note-to-Image/Video/Text tasks using a single embedding extraction, requiring 9.2$\times$ less storage and compute than the baseline CLIP-based approach. In offline large-gallery retrieval, UniNote showed a 23.5% gain in relevant sample recall over the baseline.

Significance

The paper claims that UniNote offers an efficient, practical solution for industrial-scale heterogeneous item retrieval. By unifying representation and ranking, it addresses the systemic inefficiency of decoupled pipelines and the granularity mismatch in existing MLLM embeddings. The work demonstrates that reinforcement learning can effectively align embedding spaces with content relevance, enabling high-precision ranking in a single pass. The successful deployment at Xiaohongshu validates the model's ability to balance retrieval quality with strict latency and cost constraints in real-world, large-scale applications.