Technical Summary: C-REX for Referring Expression Counting

Problem Statement

Referring Expression Counting (REC) requires models to count specific subsets of objects within an image based on fine-grained textual descriptions (e.g., "person riding a motorcycle" vs. "person walking"). The core challenge lies in distinguishing visually similar objects that differ only by subtle contextual or attribute-level cues.

Existing state-of-the-art REC methods address this via image-text contrastive learning (CL). These approaches train models to align visual tokens with positive text prompts while pushing them away from negative text prompts. However, the authors identify two critical limitations in this strategy:

1. Scarcity of Negatives: The number of available negative referring expressions for a single image is limited (typically fewer than 5), whereas contrastive learning theoretically benefits from large batches of diverse negative samples to learn robust representations.
2. Embedding Misalignment: The effectiveness of image-text CL relies heavily on the alignment between image and text embedding spaces. The authors argue that this alignment is often imperfect, leading to visual embeddings that entangle semantically distinct but visually similar objects, thereby hindering fine-grained discrimination.

Methodology: C-REX

To overcome these limitations, the authors propose C-REX (Contrastive Learning for Referring Expression Counting), a framework that shifts contrastive learning entirely into the visual embedding space, eliminating reliance on text embeddings for the contrastive objective.

Core Mechanism

Instead of contrasting image tokens against text tokens, C-REX contrasts image tokens against other image tokens within the same image.

1. Sample Definition: For a given image and referring expression (RE), the model predicts object locations. Using Bipartite Matching, predicted tokens are matched to ground-truth points.
   • Positives ($I^+$): Image tokens matched to ground-truth objects corresponding to the specific RE.
   • Negatives ($I^-$): All other image tokens in the image (including objects of the same class with different attributes, different classes, or background tokens).
2. Loss Function Modification: The authors adapt the standard Supervised Contrastive Loss [35]. A key modification is that only positive samples are used as anchors.
   • Standard supervised CL often pulls all samples of the same class together. However, in C-REX, the negative set ($I^-$) is heterogeneous (containing cars, pedestrians, background, etc.). Clustering these diverse negatives would be counterproductive.
   • Therefore, the loss function is formulated to pull positive tokens closer together while pushing them away from all negative tokens, without forcing the negatives to cluster together.
   • The objective is defined as:
     $$\mathcal{L}^*_{\text{sup}} = \sum_{i \in I^+} -\log \left( \frac{1}{|I^+|-1} \sum_{p \in I^+ - \{i\}} \frac{\exp(z_i \cdot z_p / \tau)}{\sum_{\alpha \in I - \{i\}} \exp(z_i \cdot z_\alpha / \tau)} \right)$$
3. Integration: C-REX acts as a plug-in module. It replaces the image-text contrastive loss in existing REC architectures (GroundingREC, CAD-GD) or is added alongside standard counting losses for detection-based baselines (GroundingDINO). The text input is still used to condition the visual backbone via cross-attention, but the supervision signal for discrimination comes solely from the visual space.

Extension to Class-Agnostic Counting

The framework is adapted for class-agnostic counting by defining positives based on a target class rather than a specific RE. While the improvement is less pronounced than in REC (as class-level distinctions are already easier to learn), it still yields performance gains.

Key Contributions

1. Shift to Image-Space CL: The paper demonstrates that performing contrastive learning entirely within the image embedding space is superior to image-text CL for REC, providing a more stable signal and avoiding alignment issues.
2. Scalable Negative Sampling: By utilizing unmatched visual tokens as negatives, C-REX increases the pool of negative samples from a handful of text prompts to hundreds of visual tokens per image, significantly enhancing supervision stability.
3. Plug-and-Play Framework: C-REX is designed to be architecture-agnostic, seamlessly integrating into existing REC models without structural changes.
4. State-of-the-Art Performance: The method achieves new state-of-the-art results on the REC-8K dataset across multiple backbones and architectures.

Experimental Results

The authors evaluated C-REX on the REC-8K and FSC-147 datasets using three architectures: GroundingREC, CAD-GD, and GroundingDINO.

• Performance Gains: On the REC-8K dataset, C-REX improved performance by up to 28% in Mean Absolute Error (MAE) (observed on the validation set for GroundingREC with Swin-T backbone) and 24.5% in Root Mean Squared Error (RMSE) (observed on the test set for GroundingDINO with Swin-T backbone) compared to baselines.
• Consistency: Improvements were observed across all object density ranges (0–20, 21–50, 51–100, 100+), with particular gains in fine-grained discrimination tasks (e.g., distinguishing a "blue car" from a "black car" or "person riding a bicycle" from "person riding a motorcycle").
• Ablation Studies:
  • Image-Space vs. Image-Text: Pure image-space CL outperformed both image-text CL and the combination of both, suggesting that image-text signals are suboptimal or redundant when image-space signals are optimized.
  • Positive Selection: Using Bipartite Matching to select positives outperformed strategies based on token similarity to text prompts.
  • Loss Modification: The modified loss (using only positives as anchors) significantly outperformed the standard supervised contrastive loss formulation.
• Robustness: C-REX showed improved robustness to paraphrased referring expressions compared to baselines.
• Comparison to VLMs: Specialized REC models equipped with C-REX significantly outperformed general-purpose Vision-Language Models (Molmo2-4B) in zero-shot and few-shot settings, highlighting the necessity of task-specific training for counting.

Significance and Claims

The paper claims that the "right embedding space" for contrastive learning in REC is the visual embedding space, not the image-text space. By decoupling the contrastive objective from the text modality, the model learns richer, more stable fine-grained visual representations. The authors emphasize that this approach solves the scarcity of negative samples and the instability of cross-modal alignment, leading to superior generalization in open-world counting scenarios. The framework is presented as a versatile tool that can enhance not only REC but also other counting tasks requiring fine-grained discrimination, such as class-agnostic counting.