Technical Summary: Same or Not? Enhancing Visual Perception in Vision-Language Models

Problem Statement

Current Vision-Language Models (VLMs) excel at broad visual reasoning and general scene understanding but remain coarse-grained in their perceptual abilities. They exhibit systematic biases, overlook subtle visual details, and often fail to distinguish between visually similar instances of the same object category. Existing training corpora reinforce this limitation by prioritizing general recognition (e.g., "Is it a cat or a dog?") and spatial relations over fine-grained discrimination. Consequently, models often hallucinate details or incorrectly identify distinct physical objects as identical if they share category-level semantics (e.g., brand or color), failing to attend to nuanced cues like geometry, texture, or part configuration.

Methodology

1. The TWIN Dataset

To address the lack of fine-grained supervision, the authors introduce TWIN (TWo-image INstance comparisons), a large-scale dataset containing 561,000 image-pair queries.

• Task Definition: The model is presented with two images and must determine if they depict the same physical object instance (Yes/No), rather than just the same category.
• Data Composition:
  • Instances: 1,836 unique object instances spanning 36 categories of household objects (e.g., electronics, furniture, kitchen items).
  • Positive Pairs: Images of the same physical object taken under varied viewpoints, lighting, and backgrounds.
  • Hard Negative Pairs: Distinct objects that appear visually similar (e.g., two different models of the same brand). These were curated using CLIP embeddings to find candidates and verified by human annotators to ensure they are challenging.
  • Generated Negatives: To scale the dataset, the authors used DreamBooth to synthesize variants of instances that retain overall appearance but alter fine details (e.g., removing a logo or changing a contour), creating additional hard negatives.
• Scale: The dataset comprises 22,157 unique images and scales favorably with the number of object annotations, allowing for the generation of hundreds of thousands of pairwise queries.

2. Training Paradigm

The authors post-train existing open-source VLMs (specifically Qwen2.5-VL-3B and InternVL3.5-1B) on TWIN using Reinforcement Learning (RL) with the GRPO algorithm.

• Reward Signal: The reward is binary, based solely on the correctness of the final "Yes/No" answer compared to the ground truth.
• Rationale: RL is chosen over Supervised Fine-Tuning (SFT) to prevent catastrophic forgetting and preserve generalization capabilities while instilling the specific skill of fine-grained discrimination. The model is prompted to generate a textual explanation followed by the answer, but the reward is computed only on the final classification.

3. The FGVQA Benchmark Suite

To evaluate the efficacy of TWIN, the authors introduce FGVQA (Fine-Grained VQA), a benchmark suite comprising 12,000 queries across six diverse domains:

• Domains: Retail products (ILIAS, TWIN-Eval), animals/plants (INQUIRE, CUB), landmarks, and art (MET).
• Query Types:
  • Pair: Two images, asking if they are the same instance/species.
  • Multi: One reference image and three candidates, asking how many match the reference.
• Goal: To test cross-domain generalization of fine-grained perception, ensuring models do not merely overfit to household objects.

Key Results

Performance Gains on FGVQA

Fine-tuning on TWIN yields substantial improvements in fine-grained recognition, even on domains unseen during training:

• Qwen2.5-VL (3B): Achieved a +19.3% improvement on the INQUIRE dataset (animals/plants) and +14.4% on CUB (birds). On the in-domain TWIN-Eval, performance rose from 50.1% to 67.3%.
• InternVL3.5 (1B): Despite its smaller size, it showed consistent gains, such as +3.8% on CUB and +3.0% on Landmarks.
• Comparison to Proprietary Models: While proprietary models (GPT-4o, Gemini 2.5 Flash) still lead, training on TWIN significantly narrows the gap. For instance, on INQUIRE, the gap between Qwen2.5-VL + TWIN and Gemini 2.5 Flash reduced from 29.5% to 10.2%.

Ablation Studies

• Hard Negatives: Removing hard negatives (replacing them with random negatives) resulted in a 4.5% average drop in accuracy across FGVQA, confirming that challenging, visually similar negatives are critical for learning.
• Scaling: Performance improved monotonically as the training set size increased from 5K to 561K pairs, indicating that scale is a key factor in perceptual precision.
• RL vs. SFT: Reinforcement Learning outperformed Supervised Fine-Tuning on out-of-distribution datasets (e.g., +11.1% on CUB), suggesting RL better preserves generalization while acquiring new skills.
• General Capabilities: Post-training on TWIN did not degrade performance on general VQA benchmarks (e.g., MMLU, GSM8K, TextVQA) or text-only tasks, confirming that the model retains its broad reasoning abilities.

Representation Analysis

Probing the vision encoder of the post-trained models revealed improved embeddings for fine-grained classification tasks (PETS, SUN397, CUB, CIFAR100). This suggests that TWIN training forces the underlying visual encoder to encode more discriminative, instance-level features.

Significance and Claims

The paper claims that TWIN serves as a "drop-in addition" to open-source VLM training corpora that effectively advances perceptual precision without compromising general capabilities. The authors argue that the current limitation of VLMs in fine-grained perception is largely a data problem, not an architectural one. By shifting the training objective from category-level recognition to instance-level discrimination, models learn to attend to subtle visual cues (geometry, texture, part configuration) rather than relying on coarse semantic priors.

The work establishes that:

1. Scale matters: Large-scale instance-centric datasets are necessary for robust fine-grained understanding.
2. Hard negatives are essential: Random negatives are insufficient; models require exposure to visually similar but distinct instances to learn true discrimination.
3. Generalization is possible: Training on household objects transfers effectively to art, biology, and architecture, suggesting that fine-grained perception is a generalizable skill.

The authors position TWIN and FGVQA as foundational resources for the community to measure and drive progress toward human-level visual perception in VLMs.