Technical Summary: Beyond the Literal: Decomposing Pragmatic Intent in Multimodal Meme Understanding

1. Problem Definition

Large Vision Language Models (LVLMs) currently struggle to interpret multimodal memes and sarcastic posts when the author's communicative intent diverges from the literal content. When presented with a meme (e.g., a cheerful dog with the caption "another great Monday"), LVLMs tend to generate literal descriptions of the visual and textual elements rather than inferring the underlying pragmatic meaning (e.g., complaining about work).

Existing approaches often treat intent as an end-to-end classification or generation target trained on the full fused image-text representation. This entangles the literal signal (what is depicted/said) with the pragmatic signal (why it was said). Consequently, when literal features (e.g., a smiling face or positive words) dominate the representation, the decoder defaults to "literal collapse," restating surface details regardless of the supervision label. This failure is most acute in high-divergence scenarios where the intended meaning sharply contrasts with the surface form.

2. Methodology: Intent Projection

The authors propose Intent Projection, a framework designed to structurally separate literal and pragmatic signals within a single LVLM backbone across three levels: representation, output, and objective.

A. Representation Level: Pragmatic Projection

The core innovation is an orthogonal projection module that isolates the pragmatic residual from the fused representation.

• Mechanism: The model computes a fused vector $c$ from the image and text summaries. It then identifies the "literal subspace" spanned by the unimodal summaries of the image ($u_v$) and text ($u_t$).
• Projection: The pragmatic vector $\hat{h}_{prag}$ is computed by projecting $c$ orthogonally onto the subspace orthogonal to the literal directions:
  $$\hat{h}_{prag} = (I - P_L)c$$
  where $P_L$ is the projection matrix onto the literal subspace. This removes the dominant unimodal directions, leaving only the cross-modal interaction that drives intent.
• Conditioning: $\hat{h}_{prag}$ is fed to the decoder as a soft token. Additionally, a Surface-Real Affect Classifier predicts a discrete tag from a four-way Cartesian product of surface sentiment (pos/neg) and real sentiment (pos/neg). This tag (e.g., [affect: surface-pos, real-neg] for sarcasm) serves as a persistent symbolic anchor, complementing the continuous pragmatic vector.

B. Output Level: Structured Reasoning Chain

The model is trained to externalize the decomposition as an explicit three-segment chain:

1. Literal Observation ($y_{lit}$): Describes what is shown and said.
2. Intent Inference ($y_{int}$): Resolves why the two were paired.
3. Final Intent ($y^*$): Verbalizes the author's communicative intent.

C. Objective Level: Contrastive Reinforcement Learning

Training occurs in two stages:

1. Stage 1 (SFT): Supervised Fine-Tuning teaches the chain format and affect classification. Targets are obtained via self-distillation, where the backbone generates the chain using community comments as auxiliary context (which are not visible during inference).
2. Stage 2 (GRPO): Group Relative Policy Optimization is applied with a contrastive, literal-aware reward. The reward function $R(y)$ includes:
   • A term pulling the final intent $y^*$ toward the reference intent.
   • A contrastive penalty ($-\lambda_1 \text{sim}(y^*, y_{lit})$) that explicitly pushes the final answer away from the model's own literal description.
   • Rewards for affect consistency and chain formatting.
   • Gradient Masking: Updates are applied only to $y_{int}$ and $y^*$; $y_{lit}$ is held fixed to serve as a stable reference point for the contrast.

3. Key Contributions

• Reframing the Problem: The authors identify "literal collapse" as the primary failure mode in meme understanding and reframe the task as a literal-pragmatic decomposition problem.
• Architectural Separation: They introduce a method to explicitly project out unimodal literal directions from the fused representation, granting the decoder direct access to a pragmatic residual.
• Contrastive Optimization: They propose a reward mechanism that penalizes similarity between the generated intent and the literal description, directly targeting the separation of signals rather than just task accuracy.
• Self-Distillation: The framework utilizes self-distillation to elicit intent targets from the backbone itself, avoiding the need for large-scale human annotation of intent.

4. Experimental Results

The method was evaluated across six benchmarks (MemeReaCon, MemeCap, MET-Meme, MMSD2.0, MuSE, GOAT-Bench) using three open-source backbones (LLaVA-OneVision-7B, Qwen3-VL-8B, InternVL3.5-8B).

• Performance Gains: Intent Projection consistently outperforms strong open-source baselines and narrows the gap to frontier proprietary models (e.g., Gemini-3-Flash, GPT-5.4-mini).
  • On the PI-G (Post Intent Generation) task, the 8B-scale Qwen3-VL with Intent Projection achieved 32.8% ROUGE-L, narrowing the gap to the proprietary Gemini-3-Flash by roughly 60%.
• High-Divergence Regime: The most significant improvements were observed on posts with high literal-intent divergence ($\Delta(x)$). While baseline models degraded sharply on high-divergence posts, Intent Projection maintained performance, reducing the performance gap between low and high divergence to under 5%.
• Ablation Studies:
  • Removing the contrastive reward ($\lambda_1$) caused the largest performance drop, particularly on high-divergence posts, confirming its role in preventing literal collapse.
  • Removing the affect head significantly degraded performance, validating the utility of the discrete symbolic tag.
  • The GRPO stage was found indispensable; SFT alone taught the format but failed to prevent literal collapse.

5. Significance and Claims

The paper claims that Intent Projection successfully mitigates the "literal collapse" phenomenon in LVLMs by enforcing a structural separation of signals. The authors assert that:

• Explicitly disentangling literal and pragmatic signals at the representation level is more effective than simply scaling model parameters.
• The framework generalizes the principle of literal-pragmatic separation, suggesting applicability to other tasks where speaker meaning diverges from surface form (e.g., irony in dialogue, implied stance in headlines).
• The approach bridges the gap between open-source and proprietary models specifically in scenarios requiring deep pragmatic understanding, rather than just surface-level comprehension.

Limitations Acknowledged: The authors note that their data is skewed toward English-language Western internet culture, the framework introduces marginal inference latency due to structural modifications, and the self-distillation targets rely on the latent capabilities of the backbone model.