Technical Summary: COCOLogic-V2

Problem Statement

Interpretable machine learning models, such as Concept Bottleneck Models (CBMs) and neuro-symbolic program synthesis methods, are designed to enable the verification of model decisions, a critical requirement for high-stakes domains like healthcare and finance. However, the evaluation of these models has historically been limited to narrow settings: CBMs are often assessed on single-label image classification, while program synthesis and neuro-symbolic approaches frequently rely on synthetic tasks or abstract pattern matching rather than complex inductive reasoning on real-world data.

Existing benchmarks fail to capture the complexity of visual inductive reasoning. Specifically, previous iterations of reasoning datasets (such as the original COCOLogic) often suffer from class imbalance and allow models to exploit "shortcuts" (e.g., predicting a class based on a single object like a bowl) rather than learning the underlying logical rules. Furthermore, standard evaluation metrics often mask a model's inability to handle edge cases, as high overall accuracy can be achieved by correctly classifying easy samples while failing on those that require a precise understanding of the logic.

Methodology: The COCOLogic-V2 Dataset

To address these limitations, the authors introduce COCOLogic-V2, an object-centric dataset for visual inductive reasoning on real-world images derived from MSCOCO. The dataset is designed to cover a broad subset of first-order logic, including propositional logic, object counting, counting comparisons, and object absence.

Key Design Features

1. Multilabel Framing: Unlike its predecessor, COCOLogic-V2 frames the task as multilabel classification to reduce shortcuts and class imbalance.
2. Compositional Logical Rules: The dataset comprises 10 distinct logical rules (e.g., "Signal and Ride," "Double Serving," "Herd Alone") based on the co-occurrence, absence, or specific counts of object categories within an image.
3. Fine-Grained Sample Categorization: A core innovation is the categorization of samples into three distinct types to enable automatic diagnosis of model accountability:
   • Positive Variants: Different ways a rule can be satisfied (derived from the Disjunctive Normal Form of the rule).
   • Near-Boundary (NB) Negatives: Samples that are close to the decision boundary. These are generated by flipping a single literal in the logic or modifying object counts (e.g., changing "exactly three" to "one or two"). These samples test whether a model has learned the specific details of the rule.
   • Far-From-Boundary (FB) Negatives: Samples that are easily distinguishable from positives, often classifiable without understanding the rule.
4. COCOLogic-V2-FS: A curated, small-scale version of the dataset containing 24 training samples per rule, designed specifically to evaluate few-shot and in-context learning capabilities.

Sampling Procedure

The dataset is sampled from MSCOCO images. The process involves selecting up to 1,000 images per positive variant, ensuring at least 500 images per NB type, and filling the remainder with FB samples to maintain a distribution close to the original MSCOCO dataset. The final training set contains 25,000 images, and the test set contains 3,500 images.

Experimental Evaluations

The authors evaluated several families of approaches on both the full dataset and the few-shot version:

• Concept-Based Models: Variants of CBMs using ground-truth object labels (Oracle), pretrained detectors (Mask R-CNN), supervised encoders, CLIP-based representations, Object-Centric CBMs (OCB), and the Deep Concept Reasoner (DCR).
• Black-Box Baselines: Fine-tuned ResNet-50.
• Few-Shot Approaches: In-context learning with Vision-Language Models (VLMs) such as GPT-5.5, Gemini-3.1-Pro-Preview, and Claude-Opus-4-7, as well as program synthesis using the VLP approach with Gemma-4-31B-it.

Key Results

1. Failure on Near-Boundary Samples: On the full dataset, most models achieved high overall balanced accuracy (B-Acc > 80%) by successfully separating positive samples from FB negatives. However, performance on NB samples was consistently poor, often falling to or below random guessing (e.g., 27–48% for various CBMs). This indicates that models are exploiting coarse statistical patterns rather than learning the underlying logical rules.
2. Impact of Perception Noise: Models relying on noisier concept representations (e.g., CLIP or pretrained detectors) performed worse on NB samples than those using ground-truth labels, suggesting that fine-grained rule details are difficult to capture with imperfect perception.
3. Few-Shot Challenges: In the COCOLogic-V2-FS setting, all CBM approaches struggled significantly due to the limited training signal (24 samples per rule). While in-context learning with VLMs and program synthesis performed better (reaching ~70% B-Acc), they still faced challenges. Specifically, VLMs often identified semantically related but logically imprecise patterns, and program synthesis struggled with large search spaces and identifying relevant objects without ground-truth concepts.
4. Oracle Upper Bounds: Even with perfect concept inputs (Oracle), linear predictors failed on rules involving XOR or exact counting, whereas MLPs achieved near-perfect performance, confirming the dataset is solvable in principle but requires non-linear reasoning capabilities.

Contributions

The paper makes three primary contributions:

1. COCOLogic-V2: An object-centric dataset for visual inductive reasoning over natural images that expands the logical scope to include counting and comparisons, reformulated as a multilabel task.
2. COCOLogic-V2-FS: A curated few-shot version for evaluating data-constrained settings and in-context learning.
3. Empirical Evidence of Limitations: A demonstration that current interpretable approaches (CBMs, VLMs, program synthesis) fall short in capturing underlying logical rules, particularly on near-boundary samples, highlighting visual inductive reasoning as an open challenge.

Significance and Claims

The authors claim that COCOLogic-V2 provides a concrete foundation for advancing methods in visual inductive reasoning. The dataset's fine-grained categorization (Positive, NB, FB) serves as an automatic diagnostic tool, revealing that high overall accuracy often masks a failure to learn specific logical rules.

The paper concludes that visual logical reasoning on real-world data remains an open challenge. Current models tend to separate positive and easy negative samples well but fail on samples requiring a proper understanding of rule details. The authors suggest that future work should focus on mitigating reasoning-specific shortcuts and extending datasets to include more complex logical structures and object relations to further test the limits of interpretable models.