💬 NLP

Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models

This paper identifies that dual-encoder vision-language models fail compositional reasoning due to a "Bag-of-Concepts" similarity interface, and proposes a training-free method called LCSE (Logic-Constrained Score Editing) that separates evidence extraction from constraint execution to significantly improve logical accuracy while preserving retrieval performance.

Original authors: Sultan Alshehri, Zhantao Yang, Han Zhang, Marios Savvides

Published 2026-07-28

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Sultan Alshehri, Zhantao Yang, Han Zhang, Marios Savvides

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to "see" the world by showing it millions of pictures and describing them with words. This is the world of Vision-Language Models (VLMs), a branch of artificial intelligence where computers learn to match images with text. The most popular way these robots learn is by building a shared "map" where similar pictures and similar words sit close together. When you ask the robot a question, it simply measures the distance between your words and the pictures to find the best match. It's like a giant game of "hot and cold," where the robot gets "hotter" (more confident) as it finds images that look like the words you typed.

However, there is a tricky problem with this simple distance game. While the robot is great at spotting individual things—like a "dog" or a "ball"—it often gets confused when you ask it to combine those things with logic, like "a dog but no ball" or "a cat and a bird." The robot tends to ignore the "but no" or "and" parts and just looks for the words it knows. It's as if the robot is playing a game of "Bag of Words," where it only counts how many times a word appears, completely forgetting the rules of the sentence. This paper investigates why this happens and proposes a clever new way to fix the robot's logic without having to retrain its entire brain.

The Problem: The Robot's "Bag of Concepts"

The authors of this paper discovered that standard AI models, which are supposed to be smart enough to understand complex sentences, are actually behaving like a very literal, slightly confused librarian. Imagine you ask this librarian for a book about "a red car without a driver." Instead of finding a car with no driver, the librarian hands you a book about a red car with a driver, because the librarian is so focused on the words "red" and "car" that they completely ignore the word "without."

In technical terms, these models use a "scalar similarity" score. Think of this score as a single number that tells you how well an image matches a text. The paper shows that when you ask for a combination of ideas (like "A and not B"), the model doesn't actually do the math of logic. Instead, it just averages the evidence for "A" and the evidence for "B." It's like making a smoothie: if you ask for "strawberries but no bananas," the model just blends the strawberries and bananas together and gives you a mix, because it thinks the "no" instruction is just another ingredient to be blended in, rather than a rule to remove an ingredient.

The researchers tested this by asking models to find images that matched specific logical rules, like "umbrella and no person." Even when the model could clearly see a person in the picture, it still ranked that picture as a good match because the word "person" was in the query. The model wasn't failing to see the person; it was failing to use the logic of the sentence to filter the person out.

The Investigation: Why the Logic Fails

To figure out why this happens, the authors didn't just guess; they performed a "surgery" on the model's brain. They looked inside the text the model reads to see if the logic words (like "not" or "and") were actually changing the meaning of the sentence. They found that the logic was there, hidden inside the text, but it was too weak to change the final score.

Imagine the text embedding (the digital representation of the sentence) is a team of two people carrying a heavy box. One person represents the "concepts" (the nouns like "dog" or "car"), and the other represents the "logic" (the "and" or "not"). The paper found that the "concept" person is a giant, while the "logic" person is a tiny child. When they both push the box to decide which image to pick, the giant pushes so hard that the tiny child's push is completely ignored. The result? The box moves in the direction of the concepts, and the logic gets lost in the shuffle.

The authors also checked if simply retraining the model (teaching it new examples) would fix this. They found that even after training the model on thousands of examples of "no" and "not," the model still struggled. It's like trying to teach a person who is naturally left-handed to write perfectly with their right hand by just giving them more practice; the underlying muscle memory (how the model combines scores) is the real bottleneck, not the lack of practice.

The Solution: Factored Inference and LCSE

Since the model's internal "logic muscle" is too weak to fix the problem, the authors proposed a new strategy called Factored Inference. Instead of asking the model to do the math all at once, they decided to split the job into two separate steps:

Evidence Extraction: First, ask the frozen (unchanged) model to simply identify the individual concepts. "Is there a dog? Yes, 90% sure. Is there a cat? Yes, 80% sure."
Constraint Execution: Then, take those numbers and do the logic math outside the model. If the user said "dog but no cat," the system takes the "dog" score and subtracts the "cat" score (or flips the cat score to represent "no cat") and combines them using strict logic rules.

To make this work without breaking the model's ability to handle normal sentences, they introduced a method called LCSE (Logic-Constrained Score Editing). Think of LCSE as a smart editor. It listens to the model's original answer, but if the sentence has a logic rule (like "no" or "and"), the editor steps in and tweaks the final score just enough to make the logic work. If the sentence is simple, the editor leaves the score alone. This way, the model keeps its original "personality" and retrieval skills, but it finally obeys the rules of logic.

The Results: A Big Leap in Logic

The results of this new method were impressive. The authors tested their system on a new benchmark they built called FACTOR-Bench, which is designed specifically to test if a model understands logic.

The Old Way: The best existing models (even those that had been specially trained to understand negation) got about 73.2% of the logic questions right.
The New Way (LCSE): The authors' method, using the same underlying models but with their new "editor," jumped to 85.5% accuracy.
Even Better: When they applied this method to a newer, stronger model called SigLIP 2, the accuracy soared to 90.7%.

They also tested it on a different challenge called NegBench, which focuses on "no" and "not" questions. The standard model got only 27.2% right, but with LCSE, it improved to 65.2%. Crucially, this massive improvement in logic didn't hurt the model's ability to do normal searches; it kept the same high performance for standard image retrieval.

Why This Matters

This paper suggests that the problem isn't that AI models can't "see" or "read" well enough; it's that the way they combine information is fundamentally flawed for complex tasks. By separating the act of "seeing" (finding the concepts) from the act of "thinking" (applying the logic rules), we can make these models much smarter without needing to rebuild them from scratch. It's a reminder that sometimes, the best way to fix a smart robot isn't to teach it more facts, but to give it a better calculator for the facts it already knows.

Technical Summary: Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models

1. Problem Statement

Dual-encoder Vision-Language Models (VLMs), such as CLIP and SigLIP, map images and text into a shared embedding space using a scalar similarity interface (typically cosine similarity or dot product). While highly effective for zero-shot retrieval, this interface fails to satisfy compositional constraints involving logical operators (negation, conjunction, disjunction).

The core issue is an interface-level "Bag-of-Concepts" effect. When a query contains logical operators (e.g., "umbrella and no person"), the scalar similarity score approximates a mean pooling of evidence for individual concepts rather than executing the logical operator. Consequently:

Negation fails: Queries like "no person" often retrieve images containing a person because the similarity score tracks concept presence rather than absence.
Conjunction is unstable: Queries like "A and B" frequently retrieve images containing only A or only B.
Exclusion fails: Queries aiming to exclude concepts (e.g., "neither A nor B") systematically retrieve images containing those concepts.

The authors trace this failure not to a lack of operator-dependent information in the text embeddings (which exists but is weak), but to the execution mechanism: the dot-product scoring aggregates evidence in a way that suppresses operator semantics, behaving like a soft mean-pooling operation regardless of the logical structure.

2. Methodology: Factored Inference and LCSE

The paper proposes Factored Inference, a paradigm that separates evidence extraction from constraint execution. Instead of relying on the holistic similarity of a compound query, the method extracts atomic evidence for each concept and executes logical operators externally.

2.1. Logic-Constrained Score Editing (LCSE)

The primary contribution is LCSE, a training-free method that modifies the similarity score of a frozen VLM to enforce logical constraints while preserving standard retrieval performance. The process involves four steps:

Parsing: A query is parsed into visual concepts ( $c_i$ ), their polarities (present/absent, $n_i$ ), and the logical operator ( $o$ ).
Evidence Extraction: The frozen VLM encodes the image and each atomic concept phrase independently to generate presence scores ( $p_i$ ). These scores are calibrated from raw similarity to probabilities using a sigmoid function.
Constraint Execution:
- Polarity Adjustment: For negated concepts, the score is inverted ( $1 - p_i$ ).
- Aggregation: The adjusted scores are aggregated using a power mean ( $M_\gamma$ $M_{γ}$ ).
  - Conjunction ( $A \land B$ ) uses $\gamma < 0$ (approaching min).
  - Disjunction ( $A \lor B$ ) uses $\gamma \gg 1$ (approaching max).
  - This replaces the implicit mean-pooling of the holistic score with truth-functional aggregation.
Score Editing (The "LCSE" Step): To prevent the loss of holistic information (scene context, style, spatial layout) which is not captured by atomic concept scores, the method applies a lightweight correction to the original holistic score ( $s_{hol}$ ):
$s_{LCSE} = s_{hol} + \frac{1}{\beta} \left( \text{logit}(p_{logic}) - \text{logit}(p_{soft}) \right)$
Where $p_{logic}$ is the aggregated constraint score, and $p_{soft}$ is the simple mean of concept scores. This term adjusts the holistic score only when the logical operator requires a non-mean aggregation (e.g., strict conjunction or negation). If no operators are present, the correction vanishes, and the method defaults to standard holistic retrieval.

3. Key Contributions

Mechanistic Analysis of Constraint Failure: The authors provide a "span-residual decomposition" of text embeddings, demonstrating that while operator-dependent signals exist in the residual component, they are too weak or misaligned to influence rankings under standard dot-product scoring. The span component (dominated by concept evidence) approximates mean pooling, causing the failure.
Training-Free Interface (LCSE): A method to execute Boolean constraints explicitly over atomic evidence without retraining the encoders. It preserves standard retrieval behavior where constraints do not apply.
FACTOR-Bench: A new benchmark designed to test Boolean operator semantics using operator-contrastive caption pairs (same concepts, different operators) with validated ground-truth concept presence/absence.
Empirical Validation: Extensive evaluation showing that LCSE significantly outperforms fine-tuned baselines on logical constraints while maintaining zero-shot retrieval performance.

4. Experimental Results

The authors evaluated LCSE on FACTOR-Bench, NegBench, and standard retrieval retention tasks using CLIP and SigLIP 2 backbones.

FACTOR-Bench Accuracy:
- LCSE (CLIP): Achieved 85.5% accuracy, significantly outperforming the best fine-tuned baseline (NegationCLIP at 73.2%).
- LCSE (SigLIP 2): Achieved 90.7% accuracy, demonstrating that better atomic evidence translates to better constraint execution.
- Negation Handling: On NegBench MCQ, LCSE improved accuracy from 56.2% (NegCLIP-NegFull) to 60.3% for CLIP, and from 27.2% to 65.2% for SigLIP 2.
Retrieval Retention: Unlike fine-tuned models which often degrade standard retrieval (Rank Correlation $\rho \approx 0.63\text{--}0.85$ ), LCSE preserves standard retrieval performance almost perfectly ( $\rho = 0.999$ for CLIP, R@5 unchanged).
Stratified Analysis: The performance gain of LCSE scales with the quality of atomic evidence. When concept detection is reliable (AUC > 0.90), LCSE reaches 90.4% accuracy, whereas holistic scoring achieves only 57.8%.
Ablation: The method works across different backbones and aggregation strategies. The "Score Editing" component is crucial; removing it (using factored scoring only) degrades retrieval on natural language queries that lack explicit operators.

5. Significance and Claims

The paper claims that the failure of VLMs to handle compositional logic is primarily an interface-level bottleneck (how scores aggregate evidence) rather than a representation-level failure (what encoders represent).

Modest Scope: The authors explicitly state that their analysis and method are validated on Boolean operators over visual concepts (one to two concepts, with some extension to 3-4 concepts). They do not claim to solve all compositional reasoning (e.g., attribute binding, relations, counting) but suggest that the principle of factored inference may extend if reliable atomic evidence is available.
Practical Impact: The method offers a training-free solution that can be applied to existing frozen VLMs to enable structured querying and filtering without the instability or data requirements of fine-tuning.
Limitations: The approach depends on the ability to parse Boolean structure (handled by an LLM parser with high accuracy) and the quality of the underlying encoder's atomic concept detection. If the encoder cannot detect a concept reliably, external execution cannot recover correctness.

In summary, the paper argues that "Similarity Is Not Logic" and proposes Factored Inference as a mechanism to decouple evidence extraction from logical constraint execution, achieving high accuracy on logical tasks while maintaining the robust zero-shot retrieval capabilities of standard VLMs.