🤖 AI

Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning

The paper proposes Text as Partial Constraint (TPC), a core-residual alignment framework that treats multi-view captions as incomplete supervision to distill a consensus semantic core and explicitly discourage reliance on unsaid residuals, thereby achieving robust and reliable vision-language representations under underspecified language supervision.

Original authors: Chengzhen Yu, Canran Xiao, Siyuan Ma, Yang Liu

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chengzhen Yu, Canran Xiao, Siyuan Ma, Yang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Over-Confident" Translator

Imagine you are trying to teach a robot to recognize a dog in a park. You show it a picture and give it a description: "A golden retriever running in the park on a sunny day with a ball."

The robot learns to match the picture to that specific sentence. But here is the catch: Human descriptions are messy and incomplete.

If you show the robot the same picture but with a slightly different description—"A dog running in the park"—a standard AI might get confused. It might think, "Wait, the first one said 'golden retriever' and 'sunny day,' but this one didn't. Is it a different dog? Is it raining now?"

Because the robot was trained to treat every single sentence as the complete truth, it gets "over-confident" about details that weren't actually in the picture (like the specific breed or the weather) just because the sentence mentioned them. This makes the robot brittle; if you change the words slightly (paraphrase) or leave out a detail, the robot's confidence crashes, or it makes wild guesses (hallucinations).

The Solution: TPC (Text as Partial Constraint)

The authors propose a new way to train these robots called TPC. Instead of treating every sentence as the whole truth, they treat every sentence as a partial clue.

Think of it like a group of detectives looking at the same crime scene photo, but each detective writes a different report:

Detective A: "A dog running on grass."
Detective B: "A brown animal moving fast near a tree."
Detective C: "A pet running in a park."

The Old Way: The robot tries to memorize every single detail from every report. It gets confused when the reports disagree (e.g., "Is it brown or golden?").

The TPC Way: The robot looks for the Consensus Core. It asks: "What do ALL these reports agree on?"

Agreed Core: "A dog running in a park." (This is the truth).
The "Unsaid" Residual: "Golden," "Sunny," "Brown," "Tree." (These are details that only some reports mentioned, or details that might be missing).

TPC teaches the robot to:

Lock onto the Core: Focus only on the parts all descriptions agree on.
Ignore the "Unsaid": Actively forget the parts that are specific to just one description.
Admit Uncertainty: If the descriptions disagree a lot (e.g., one says "dog," another says "cat"), the robot should lower its confidence instead of guessing wildly.

How It Works (The "Core-Residual" Filter)

The paper introduces a clever mechanism to make this happen:

The Core Filter: Imagine a sieve. When a new sentence comes in (even if it's just one sentence, like "A dog running"), the sieve filters out the "fluff" (the unique, view-specific details) and keeps only the "nutritious core" (the shared meaning).
The Residual Penalty: The robot is punished if it tries to match the picture to the "fluff." If the picture doesn't clearly show a "sunny day," but the sentence says it, the robot learns not to get excited about that part. It learns to say, "I see the dog, but I'm not sure about the weather, so I won't bet my whole answer on it."
Calibrated Confidence: If the robot sees a sentence that is very vague or disagrees with other potential descriptions, it automatically becomes more "cautious." It doesn't say "I am 100% sure!" It says, "I'm pretty sure, but let's be careful."

Why This Matters (The Results)

The paper tested this idea on standard AI benchmarks (like ImageNet and various retrieval tasks). Here is what they found:

Stronger Robustness: When the descriptions were changed slightly (paraphrased) or details were removed, the TPC robot didn't crash. It stayed stable because it was focusing on the core truth, not the fluff.
Less Hallucination: When used inside larger AI systems (Large Vision-Language Models), the robot made fewer mistakes where it "saw" things that weren't there. It stopped confidently guessing details that weren't supported by the text.
Better Accuracy: Even though it was ignoring some details, it actually got more questions right because it stopped getting distracted by unreliable information.

The Bottom Line

The paper argues that language is naturally incomplete. We don't always say everything we see.

Previous AI models tried to memorize every word we said, which made them fragile. TPC teaches AI to understand that a sentence is just a partial constraint—a hint, not a full blueprint. By focusing on what is consistently true across different ways of describing the same thing, and ignoring the "noise" of specific details, the AI becomes more reliable, less over-confident, and much harder to trick.

In short: TPC teaches the AI to stop guessing the weather just because one person mentioned it, and instead focus on the fact that there is a dog in the park.

Technical Summary: Text as Partial Constraint (TPC)

Problem Statement
Vision-language alignment underpins open-vocabulary recognition, retrieval, and Large Vision-Language Models (LVLMs). However, natural language supervision is inherently incomplete: captions often omit details, rely on shared context, or vary significantly in phrasing. Existing methods typically treat each caption as a complete target, forcing models to align with view-specific details that may be absent in other valid descriptions. This leads to "brittle" matching (where performance drops under paraphrase), "over-confident" predictions (where the model is highly certain even when the text constraint is weak), and downstream hallucinations in LVLMs. The paper identifies that current models over-commit to "unsaid" residuals—view-specific deviations that are not consistently supported across multiple caption views of the same image.

Methodology: Text as Partial Constraint (TPC)
The authors propose TPC, a core–residual alignment framework that reframes natural captions as partial constraints rather than complete targets. The method operates on the premise that multiple caption views of an image share a stable "consensus core" while diverging in an "unsaid" residual space.

The framework consists of four key components:

Multi-View Consensus Core: For an image with $K$ caption views, a consensus core $c_i$ is defined as the normalized mean direction of the text embeddings. This core represents the shared, view-invariant semantics.
Single-View Core Filter: To enable deployment with a single query text, a lightweight linear filter $\psi_\theta$ is trained to predict the consensus core $c_i$ from a single view embedding $t_{i,k}$ . This ensures the model can recover the robust semantic target during inference without requiring multiple captions.
Unsaid Residual Suppression: The text embedding is decomposed into the predicted core and an orthogonal "unsaid" residual $r_{i,k}$ . A non-commitment regularizer explicitly penalizes the alignment between the image embedding and these residuals, preventing the model from over-fitting to view-specific details.
Uncertainty-Aware Contrastive Alignment: An uncertainty metric $u_i$ is computed based on the disagreement (variance) among the $K$ caption views relative to the consensus. This uncertainty modulates the temperature $\tau_i$ in the InfoNCE loss. When captions disagree (high uncertainty), the alignment is "softened" to reduce overconfident updates; when they agree, the alignment is sharpened.

The total training objective combines the core alignment loss, an agreement loss to ensure the single-view filter matches the consensus, and the non-commitment loss to suppress residuals.

Key Contributions

Formalization of Language Underspecification: The paper formalizes caption-view variation as a form of language underspecification and identifies over-commitment to "unsaid" residuals as a primary mechanism behind brittleness and hallucination.
Core–Residual Framework: It introduces a training principle that prioritizes view-invariant semantics and produces confidence scores that track the strength of text constraints, while remaining deployable with a single query.
Theoretical Justification: The authors provide theoretical bounds showing that robustness improves by increasing core alignment and decreasing residual sensitivity. They also demonstrate that multi-view consensus acts as a denoising mechanism for incomplete textual supervision, with statistical guarantees on generalization.

Experimental Results
TPC was evaluated across zero-shot recognition, adversarial robustness, and LVLM transfer:

Zero-Shot Recognition: On ImageNet, TPC achieved 81.42% clean and 64.05% robust Top-1 accuracy, outperforming the strongest robust baseline (Adv-W2S) by +5.58/+5.75. On the Avg-14 transfer suite, it achieved 76.19% clean and 52.03% robust accuracy.
LVLM Transfer: When used as a vision encoder for LLaVA-1.5-7B, TPC improved object hallucination metrics, achieving an average 85.16 POPE F1 score (surpassing Adv-W2S by +1.36) and 59.57 OKVQA accuracy.
Robustness Analysis: TPC demonstrated reduced sensitivity to caption choice (lower rank volatility) and better calibration under weak text constraints. In stress tests involving controlled caption deletions, TPC maintained higher accuracy and lower Expected Calibration Error (ECE) compared to standard CLIP fine-tuning.

Significance and Claims
The paper claims that modeling text as a partial constraint is a practical and principled route to more reliable vision-language representations. By explicitly separating the stable semantic core from the unstable, view-specific residuals, TPC addresses the root cause of brittleness in underspecified language supervision. The authors emphasize that this approach does not require changing the architecture of downstream LVLMs but rather provides a more robust "perceptual backbone" that reduces over-commitment to weak signals. The method is presented as a drop-in replacement for vision encoders that complements existing decoding- and connector-level fixes for hallucination.