Technical Summary: Omni-Persona

Problem Statement

While multimodal large language models (MLLMs) have advanced significantly in processing text, images, and audio, research into omnimodal personalization remains fragmented. Existing approaches are predominantly limited to vision-language settings, neglecting audio as a primary persona modality. Furthermore, current evaluation protocols suffer from three critical gaps:

1. Lack of Unified Coverage: Benchmarks rarely integrate text, image, and audio simultaneously, often treating audio as secondary or excluding it entirely.
2. Idealized Evaluation Settings: Most benchmarks assume the target persona is always present in the retrieved context (answerable-only), failing to account for real-world retrieval noise where the queried identity may be absent. This leads to a failure to detect "hallucination" (answering when the persona is missing) or "over-abstention" (failing to answer when the persona is present).
3. Insufficient Diagnostic Rigor: Standard recall metrics conflate grounding ability with the model's willingness to answer, obscuring calibration failures. There is a lack of systematic study on how post-training regimes (Supervised Fine-Tuning vs. Reinforcement Learning) affect omnimodal grounding and calibration.

Methodology: Omni-Persona Benchmark

The authors introduce Omni-Persona, the first comprehensive, evaluation-only benchmark for omnimodal personalization. The methodology is built on the following components:

1. Formalization: Persona Modality Graph (PMG)

The task is formalized as a cross-modal routing problem over a Persona Modality Graph ($G = (V, E)$).

• Nodes: Represent a query ($Q$) and retrieved context nodes ($C_1, \dots, C_K$). Each node contains a triplet of modalities: visual ($v$), audio ($a$), and textual ($t$).
• Task: The model must identify the relevant context node $C_j$ corresponding to the query cue and ground its response in the associated text $t_j$. If the target persona is absent from all contexts, the model must abstain.
• Raw Context: Unlike prior works that compress multimodal signals into text, Omni-Persona utilizes raw-form contexts (original images and audio clips) to preserve fine-grained identity cues (e.g., vocal timbre, facial geometry) that are difficult to verbalize.

2. Benchmark Design

• Scale: ~750 evaluation items spanning 4 task groups and 18 fine-grained tasks.
• Task Groups:
  1. I2I (Image-to-Image): Visual identity recognition.
  2. A2A (Audio-to-Audio): Voice identity recognition.
  3. T2T (Text-to-Text): Same-modal semantic retrieval.
  4. T2Any (Text-to-Any): Cross-modal semantic grounding (e.g., text query to audio/image context).
• Absent-Persona Evaluation: Approximately 50% of samples are "no-GT" (ground-truth absent), requiring the model to perform structured abstention rather than hallucinating an answer. Hard distractors (visually or vocally similar identities) are included to simulate retrieval noise.

3. Evaluation Metrics

To address the limitations of recall-only metrics, the authors propose Calibrated Accuracy (Cal):
$$Cal = \frac{n_a \cdot Ans + n_u \cdot Unans}{n_a + n_u}$$
Where:

• $Ans$: Answerable Recall (correctness on answerable items).
• $Unans$: Unanswerable Recall (correct abstention on absent-persona items).
• Anti-Hallucination Metrics: $1-FA$ (False Abstention rate on answerable items) and $TA$ (True Abstention rate on unanswerable items) are used to diagnose specific failure modes (over-abstention vs. hallucination).

4. Post-Training Analysis

The paper investigates two post-training regimes on open-source backbones (Qwen2.5-Omni, Gemma4):

• Supervised Fine-Tuning (SFT): Trained on 1K and 10K datasets of curated ground-truth annotations.
• Reinforcement Learning with Verifiable Rewards (RLVR): Trained without SFT warm-up using three verifiable reward components:
  1. Perception: Rule-based binary matching (yes/no) for visual/audio identity.
  2. Location: Rule-based selection of the correct context index.
  3. Retrieval: LLM-as-a-Judge verification of grounded answers or correct abstention.

Key Results

Experiments were conducted on closed-source (Gemini-3 family) and open-source models (Qwen, Gemma, MiniCPM, Phi, Interactive-Omni).

1. Audio-Visual Grounding Gap: Open-source models exhibit a significant performance gap between visual (I2I) and audio (A2A) grounding, with audio recall lagging by 5–35 percentage points. While RLVR improves audio recall, it does not fully close the gap or resolve calibration issues.
2. Recall and Scale are Incomplete Diagnostics: Larger models (e.g., Qwen3-30B) do not necessarily achieve higher Cal scores than smaller variants (e.g., Qwen2.5-3B). High recall often coexists with low True Abstention (TA), indicating that scaling alone does not solve hallucination in absent-persona scenarios.
3. SFT Limitations: Scaling SFT data from 1K to 10K samples failed to consistently improve Cal scores and sometimes degraded performance. This suggests that constructing high-quality, in-domain ground-truth supervision for diverse open-ended scenarios is a bottleneck that SFT cannot easily overcome.
4. RLVR Trade-offs: RLVR significantly improved Answerable Recall (up to +9.2% on Qwen-7B) by directly optimizing for perceptual matching and grounded retrieval. However, it induced a strong answer-bias: models became less likely to abstain, causing TA to drop significantly (e.g., -15.4% on Qwen-7B). Consequently, Calibrated Accuracy (Cal) for RLVR-trained models remained at or below the base model levels, revealing a recall-calibration trade-off.

Significance and Contributions

The paper claims the following contributions:

1. Omni-Persona Benchmark: The first comprehensive benchmark for omnimodal personalization that unifies text, image, and audio modalities and treats absent-persona cases as a core evaluation dimension via the Persona Modality Graph.
2. Calibrated Accuracy Metric: A novel evaluation framework that jointly assesses grounding correctness and appropriate abstention, exposing failure modes (hallucination and over-abstention) masked by traditional recall-only protocols.
3. Diagnostic Insights: The study provides a model-specific diagnostic map revealing:
   • A consistent visual-over-audio grounding asymmetry in open-source models.
   • The insufficiency of recall and parameter scale as standalone metrics for personalization.
   • Distinct failure modes of SFT (limited by data construction scalability) and RLVR (tendency toward answer-bias and reduced calibration).

The authors conclude that Omni-Persona serves as a diagnostic framework to guide future research in post-training and reward design, highlighting that current methods struggle to balance high recall with calibrated abstention in realistic, noisy retrieval scenarios.