💬 NLP

Estimating Commonsense Plausibility through Semantic Shifts

This paper introduces ComPaSS, a novel discriminative framework that estimates commonsense plausibility by measuring semantic shifts induced by augmentations, demonstrating superior performance over generative baselines across language and vision-language models while highlighting the benefits of contrastive pre-training.

Original authors: Wanqing Cui, Wei Huang, Keping Bi, Jiafeng Guo, Xueqi Cheng

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wanqing Cui, Wei Huang, Keping Bi, Jiafeng Guo, Xueqi Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Why AI Gets "Common Sense" Wrong

Imagine you are teaching a robot how to understand the world. You tell it, "A penguin is a bird." The robot learns this perfectly. But then you ask, "Is a penguin green?"

A human knows immediately: No, that's silly. Penguins are black and white.
But a standard AI (Large Language Model) might hesitate. Why? Because it learned from reading billions of sentences on the internet. It might have read a story about a "green penguin" in a cartoon, or it might just be guessing based on how often the words "penguin" and "green" appear near each other. It struggles to distinguish between what is statistically likely to be written and what is actually true in the real world.

Existing methods try to fix this by asking the AI to "guess" the answer or calculate the probability of the sentence. But the authors of this paper say: "Stop guessing. Start comparing."

The Solution: ComPaSS (The "Reality Check" Tool)

The authors propose a new method called ComPaSS. Instead of asking the AI to generate an answer, they treat it like a detective checking for "semantic shifts" (changes in meaning).

Here is the core idea, explained with a simple analogy:

The Analogy: The "Smoothie Test"

Imagine you have a glass of plain water (the Anchor Sentence).

Scenario A: You add a drop of blue food coloring. The water turns slightly blue. The change is small and natural.
Scenario B: You dump a whole bucket of concrete into the water. The water turns into a thick, gray sludge. The change is massive and unnatural.

ComPaSS works like this:

The Anchor: It takes a basic sentence, like "There is a penguin."
The Candidate: It adds a specific detail to make a new sentence, like "There is a green penguin."
The Measurement: It measures how much the "meaning" of the sentence shifted.
- If the AI says, "Wait, penguins aren't green," the meaning shifts a lot. The "flavor" of the sentence changes drastically. This means the idea is implausible.
- If the AI says, "Yes, penguins are black," the meaning shifts very little. The sentence still feels like "penguin water." This means the idea is plausible.

The Rule: The less the meaning changes when you add a detail, the more "common sense" that detail has.

Why This is Better Than Old Methods

The paper compares ComPaSS to two other ways AI usually tries to judge truth:

The "Likelihood" Method (The Coin Flipper):
- How it works: It asks, "How likely is the AI to type the word 'green' after 'penguin'?"
- The Flaw: If the AI has read a lot of funny stories about green penguins, it might think "green" is likely, even though it's false. It confuses popularity with truth.
The "Verbalization" Method (The Interviewer):
- How it works: It asks the AI, "Is a green penguin plausible? Yes or No?"
- The Flaw: The AI might get confused by its own confidence. It might say "Yes" just to sound helpful, or it might overthink the question.

ComPaSS (The "Reality Check"):

It doesn't ask the AI to guess or talk. It simply measures the distance between the two sentences in the AI's "mind."
It's like checking if a puzzle piece fits. You don't ask the puzzle, "Do you think this fits?" You just try to push it in. If it fits smoothly (small shift), it's a match. If it forces the puzzle apart (big shift), it's wrong.

The Secret Weapons: Eyes and Contrast

The paper found two special ingredients that make ComPaSS work even better:

1. Vision-Language Models (VLMs) = The "Eyes"

Text-only AI models (like standard LLMs) suffer from "Reporting Bias."
- Analogy: Imagine a news report that only mentions "black sheep" because it's a famous idiom. The reporter never mentions "white sheep" because it's boring. The AI thinks black sheep are common because it only reads about them.
VLMs (models that can see images) fix this. They look at a picture of a sheep. They see it's white. They know the text report was biased.
Result: ComPaSS with "eyes" (VLMs) is much better at judging things like colors and shapes than text-only models.

2. Contrastive Pre-training = The "Sharpener"

Imagine a student who is good at math but bad at spotting tiny differences between numbers.
Contrastive training is like a special exercise where the student is forced to practice spotting the difference between "very similar" and "very different" things.
Result: When you use a model that has done this "sharpness training," ComPaSS can detect even the tiniest, most subtle common sense errors that other models miss.

The Results: Who Won?

The researchers tested this on many tasks, like:

"What color is a penguin?" (Red, Green, Black, Gray?)
"Where are farmers with new crops?" (In a truck? At a farm?)

The Winners:

ComPaSS beat almost every other method, including the famous GPT-4.
VLMs (models with eyes) were the champions for visual tasks (colors, shapes).
Smaller models (with fewer parameters) using ComPaSS often beat massive models using old methods. This proves that how you measure the answer is more important than just making the AI bigger.

Summary

ComPaSS is a new way to test if an AI understands common sense. Instead of asking the AI to guess or talk, it measures how much the "meaning" of a sentence breaks when you add a weird fact.

Small break? The fact is true (e.g., Penguins are black).
Big break? The fact is nonsense (e.g., Penguins are green).

By using this "semantic shift" measurement, especially with models that can see images, we can build AI that understands the real world much better than just reading books.

1. Problem Statement

Commonsense Plausibility Estimation (CSPE) is the task of evaluating whether a statement or an augmentation to a statement aligns with human commonsense knowledge. While Large Language Models (LLMs) have advanced text generation, ensuring their outputs are commonsense-plausible remains a challenge.

Existing approaches to CSPE rely on generative perspectives, which fall into two categories:

Likelihood-based: Using token prediction probabilities (perplexity) to judge plausibility.
Verbalization-based: Prompting models to generate natural language judgments (e.g., "Is this plausible? Yes/No").

Limitations of Current Methods:

Fine-grained discrimination: Generative methods struggle to distinguish between subtle differences in plausibility (e.g., ranking "black sheep" vs. "gray sheep" vs. "red sheep").
Reporting Bias: LMs trained on text often suffer from statistical biases where common idioms (e.g., "black sheep") are overrepresented, leading models to incorrectly infer that the attribute is physically common rather than linguistically frequent.
Task Mismatch: CSPE is inherently a discriminative task (ranking options), yet current methods often treat it as a generative task.

2. Methodology: ComPaSS

The authors propose ComPaSS (Commonsense Plausibility through Semantic Shifts), a discriminative, zero-shot framework that quantifies plausibility by measuring the semantic shift caused by adding commonsense information.

Core Concept

The underlying hypothesis is: Plausible augmentations induce minimal semantic shifts, while implausible ones cause substantial deviations.

Example: Adding "black" to "There is a penguin" causes a small semantic shift (plausible). Adding "green" causes a large shift (implausible).

Workflow

Sentence Construction:
- Anchor Sentence ( $s_{anchor}$ ): Contains the base context but omits the specific commonsense detail (e.g., "There is a penguin").
- Candidate Sentence ( $s_{candi}$ ): Augments the anchor with the candidate attribute (e.g., "There is a black penguin").
- Adaptability: For structured inputs (triplets), templates are used. For free-form inputs (questions), an LLM (GPT-4) generates coherent sentences.
Semantic Representation:
- Both sentences are encoded into dense vector representations ( $r_{anchor}$ and $r_{candi}$ ) using a pre-trained backbone model ( $\theta$ ).
- The backbone can be a Language Model (LM) (e.g., RoBERTa, Mistral) or a Vision-Language Model (VLM) (e.g., CLIP, EVA-CLIP).
- The method is architecture-agnostic, using specific tokens (e.g., [CLS], [EOS], or last token) to derive sentence embeddings.
Scoring via Semantic Shift:
- Plausibility is calculated as the semantic similarity between the anchor and candidate embeddings.
- Formula: $p_i \propto \text{sim}(r_{anchor}, r_{candi})$ .
- Higher similarity = Lower semantic shift = Higher plausibility.
- Candidates are ranked based on these scores.

Key Enhancements

Backbone Selection: The framework tests both LMs and VLMs. VLMs are hypothesized to mitigate reporting bias by incorporating visual grounding (e.g., knowing penguins are black via images, not just text statistics).
Contrastive Pre-training: The authors utilize models pre-trained with contrastive learning (e.g., Sup-SimCSE, E5) to sharpen the model's ability to distinguish semantic nuances, which is critical for measuring small shifts.

3. Key Contributions

Novel Discriminative Framework: ComPaSS shifts the paradigm from generative likelihood/verbalization to discriminative semantic shift measurement, proving more effective for fine-grained ranking.
Modality Analysis: Demonstrates that Vision-Language Models (VLMs) significantly outperform text-only LMs on vision-grounded commonsense tasks (e.g., object colors, shapes) by overcoming textual reporting bias.
Contrastive Learning Impact: Shows that backbones enhanced with contrastive pre-training yield superior results, as they create sharper semantic boundaries necessary for detecting subtle plausibility differences.
Generalizability: The method works across diverse input formats (structured triplets vs. free-form text) and knowledge levels without requiring task-specific fine-tuning.

4. Experimental Results

The method was evaluated on two fine-grained CSPE tasks:

Attribute Value Ranking: CoDa (Color) and ViComTe (Color, Shape, Material).
Commonsense Frame Completion (CFC): Ranking answers to open-ended questions.

Key Findings:

Outperformance: ComPaSS consistently outperformed baselines (including likelihood-based and verbalization-based methods) across all backbones.
- Example: On CoDa (Color), ComPaSS with EVA-CLIP achieved 62.87% Spearman correlation, significantly beating the best baseline (VERA-T5 at 58.93%).
VLM Superiority: VLMs (EVA-CLIP, CLIP) significantly outperformed large LLMs (Mistral-7B, Qwen2-7B) on visual attributes. This confirms that visual grounding is crucial for correcting text-based biases.
Contrastive Learning: Models with contrastive pre-training (e.g., RoBERTa w/ CL, Mistral w/ CL) consistently beat their vanilla counterparts.
Closed-Source Comparison: ComPaSS (using open-source backbones) outperformed GPT-4 on several binary classification tasks, highlighting the limitations of GPT-4's verbalization approach for fine-grained ranking.
Parameter Efficiency: Smaller encoder-only models (e.g., RoBERTa-355M) combined with ComPaSS achieved results comparable to or better than billion-parameter decoder-only models.

5. Significance and Implications

Paradigm Shift: The paper argues that CSPE is fundamentally a discriminative task. Using semantic similarity (discriminative) is more robust than probability estimation or verbal generation (generative) for ranking plausibility.
Addressing Bias: The study provides empirical evidence that multimodal learning (VLMs) is essential for correcting "reporting bias" in language models, where linguistic frequency is mistaken for physical reality (e.g., the "black sheep" idiom vs. actual sheep colors).
Efficiency: ComPaSS offers a parameter-efficient pathway for high-quality commonsense evaluation, leveraging existing pre-trained embeddings without the need for massive fine-tuning datasets.
Future Direction: The results suggest that future commonsense reasoning systems should prioritize semantic representation quality (via contrastive learning) and multimodal grounding over simply scaling up generative parameters.

6. Limitations

Absolute Judgments: ComPaSS provides comparative rankings (relative plausibility) rather than absolute pointwise scores. Establishing absolute thresholds for plausibility remains difficult.
Bias Inheritance: Like all models, ComPaSS inherits biases from its training data (LMs/VLMs). If the training data contains skewed attribute distributions, the model may perpetuate these biases in its rankings.