Technical Summary: RoCo-ACE

Problem Statement

Multimodal Large Language Models (MLLMs) require updates to absorb new factual or domain-specific knowledge after deployment. However, standard knowledge injection strategies face a fundamental injection-retention trade-off:

• Direct Fitting (SFT/LoRA): Optimizing directly on authoritative answers improves injected-knowledge accuracy but often causes significant drift in non-updated behaviors (e.g., visual grounding, safety, general reasoning) due to full-answer imitation and style overfitting.
• Constrained Updates: Methods that restrict the update space (e.g., LoRA variants, null-space constraints) limit drift but often fail to localize supervision to specific factual units, resulting in under-injection of target facts.
• Online Distillation: Training on model-generated rollouts rather than fixed targets keeps updates close to the current policy. However, uniform reference-conditioned distillation provides coarse supervision. It allocates gradient mass indiscriminately to generic wording, non-target visual details, and answer style, while under-emphasizing tokens supported by the reference and providing only indirect supervision for facts omitted from the current rollout.

The core challenge is achieving fine-grained supervision that updates specific factual anchors without forcing the model to clone the entire reference answer or degrade its general capabilities.

Methodology: RoCo-ACE

The authors propose RoCo-ACE, a rollout-conditioned online distillation objective designed to inject sparse factual content while preserving retention. The method operates on a dataset of prompts ($x_i$), images ($I_i$), and authoritative reference answers ($a_i$). For each example, the student model $p_\theta$ generates an on-policy rollout $y_i$.

The objective combines two complementary components:

1. RoCo: Rollout-Conditioned Contrastive Online Distillation

RoCo addresses the supervision granularity issue for tokens already present in the student's rollout. It utilizes a teacher model $q_\phi$ (an exponential moving average of the student) to score the same rollout prefix under two conditions:

• Base View ($q_{base}$): Prompted with the user input and image only.
• Reference View ($q_{ref}$): Prompted with the user input, image, and the authoritative reference answer.

The method computes a likelihood contrast ($\Delta_{i,t}$) for each token $t$ in the rollout:
$$\Delta_{i,t} = \log q_{ref}(y_{i,t} | \dots) - \log q_{base}(y_{i,t} | \dots)$$
Tokens with a positive contrast (i.e., their likelihood increases significantly when the reference is available) are identified as reference-supported. RoCo reallocates additional distillation weight to these tokens, while generic or unsupported tokens receive only a "floor" distillation signal. This ensures the model learns to align specific factual spans with the reference without forcing uniform imitation of the entire response style.

2. ACE: Anchored Cross-Entropy

RoCo cannot supervise facts that are omitted from the student's rollout. ACE addresses this coverage gap by applying sparse supervised correction to extracted authoritative anchors (e.g., entities, dates, numbers) from the reference answer.

• Coverage Scoring: The system calculates a coverage score $\rho_i(s, y_i)$ for each factual span $s$ in the reference against the student rollout.
• Weighted Correction: Spans that are missing or weakly covered receive higher weights. The loss is computed as a weighted cross-entropy over only the anchor tokens in the reference:
  $$L_{ACE} = - \sum_{k \in A_i} u_{i,k} \log p_\theta(a_{i,k} | \dots)$$
  where $u_{i,k}$ is the anchor weight. This provides direct supervision for missing facts without treating the full reference answer as a token-level imitation target.

Overall Objective

The final loss combines the weighted rollout distillation and the sparse anchor correction:
$$L = \frac{1}{|B|} \sum_{i \in B} (\lambda_{RoCo} L_{RoCo}^{(i)} + \lambda_{ACE} L_{ACE}^{(i)})$$

Key Contributions

1. Identification of Supervision Granularity: The paper identifies that uniform online distillation fails to distinguish between reference-supported factual tokens and generic wording, leading to suboptimal injection and retention.
2. RoCo-ACE Framework: The introduction of a dual-objective system:
   • RoCo: Uses same-rollout likelihood contrast to reweight reference-supported tokens, strengthening factual alignment on-policy.
   • ACE: Adds sparse anchored correction for omitted authoritative facts, ensuring coverage without full-answer cloning.
3. Empirical Validation: Extensive experiments across three knowledge-injection settings (EVOKE, VP, Sci) and six retention benchmarks demonstrate that RoCo-ACE achieves superior injected-knowledge accuracy while maintaining retention levels close to the base model and online-distillation baselines.

Experimental Results

The authors evaluated RoCo-ACE using Qwen3-VL-8B (and Qwen3-VL-30B in appendices) against baselines including SFT, LoRA, KORE, SDFT, and various constrained-update methods.

• Injection Accuracy: RoCo-ACE achieved the highest injected-knowledge accuracy across all settings. On the EVOKE benchmark, it reached 27.6% accuracy, outperforming the strong constrained baseline KORE (25.2%) and the direct-fitting SFT (20.1%).
• Retention: Crucially, RoCo-ACE maintained a Ret. Avg. (average retention across six benchmarks) of 56.5%, which is significantly higher than SFT (31.8%) and KORE (47.0%), and comparable to the base model (57.2%) and the online distillation baseline SDFT (55.1%).
• Ablation Studies:
  • Removing the contrastive weighting (RoCo) and using uniform weights reduced injection accuracy.
  • Removing the anchor correction (ACE) resulted in lower injection accuracy, particularly for omitted facts.
  • Using full-reference Cross-Entropy (SFT) alongside distillation improved injection less effectively than ACE and degraded retention more severely, confirming that sparse anchor correction is superior to full-answer imitation.
• Qualitative Analysis: Case studies showed that RoCo emphasizes useful on-policy content (e.g., "street art," "Amsterdam") while ACE corrects specific missing anchors (e.g., "STRAAT Museum," specific dates) that the rollout initially omitted.

Significance and Claims

The paper claims that RoCo-ACE effectively resolves the injection-retention trade-off by shifting the focus from behavior cloning to factual alignment. By decoupling the supervision of reference-supported rollout tokens from the correction of omitted anchors, the method injects sparse factual content without turning the full reference answer into a behavior-cloning target.

The authors position RoCo-ACE as a complementary approach to parameter-space constraints; rather than restricting where the model can change, it improves how the model is supervised during the update. The method is designed specifically for settings where updates are accompanied by authoritative natural-language references, making it suitable for entity, event, and domain-specific knowledge injection in MLLMs. The paper modestly notes that retention is evaluated empirically on held-out benchmarks and does not provide formal guarantees for all non-updated capabilities.