💻 computer science

CRISP: Compositional Relations as Invariant Structural Priors for Domain Generalization

The paper proposes CRISP, a novel image classification framework that enhances domain generalization by explicitly modeling visual recognition as the composition of spatial primitives and their invariant relational structures, achieving state-of-the-art performance across diverse real-world datasets.

Original authors: Dat Nguyen, Duc-Duy Nguyen

Published 2026-09-24✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dat Nguyen, Duc-Duy Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of computer vision, machines are trained to recognize objects by studying thousands of examples. A computer learns what a dog looks like by seeing many photos of dogs. However, these machines often struggle when the conditions change. A dog photographed in bright sunlight, a sketch of a dog, or a cartoon drawing of a dog might look so different to the machine that it fails to recognize the animal, even though a human would see them as the same thing. This problem is known as domain generalization. The goal is to build a system that learns the essential, unchanging truths about an object so it can identify it correctly, no matter how the lighting, style, or background shifts. Researchers have long suspected that the secret to this stability lies not just in the colors or textures of an object, but in how its parts are arranged relative to one another.

A team of researchers has developed a new approach called CRISP to solve this problem. Instead of trying to force a computer to memorize every possible variation of a dog or a bird, the system breaks the image down into its fundamental building blocks. Imagine a bird not as a single, complex image, but as a collection of simple parts: a beak, eyes, wings, and feet. The CRISP system first learns to find these specific parts in any image, regardless of whether the image is a photograph, a painting, or a rough sketch. It then focuses on the relationships between these parts. It learns that for a bird to be a bird, the beak must be near the eyes, and the wings must sit on the sides of the body. These spatial relationships are the stable rules that remain true even when the visual style changes completely.

The researchers tested this idea by training the system on a variety of datasets where the visual style shifted dramatically. They used five different real-world image collections that varied in how the subjects were depicted, where the photos were taken, and what kind of cameras were used. They also tested it on a specific challenge involving birds, where the difference between species often comes down to the precise combination of small features like beak shape and wing patterns. In these tests, the CRISP system significantly outperformed existing methods. On the general collection of images, it improved accuracy by 1.2 percentage points over the previous best method. On the bird recognition test, the improvement was even more dramatic, jumping by 7.9 percentage points. This suggests that explicitly teaching the computer to look for the arrangement of parts, rather than just the overall look of the object, is a powerful way to make it more reliable.

To achieve this, the system uses a three-step process that works together seamlessly. First, it scans the image to find the locations of the visual primitives, or the basic parts. It does not need a human to draw boxes around these parts beforehand; the system learns to identify them on its own. Second, it measures the distance and angle between these parts using flexible rules that can adapt to different shapes. For instance, it can determine if one part is above another, or if three parts form a specific triangular shape. Third, it combines this structural information with the local appearance of the parts to make a final decision. If the system sees a beak and eyes in the right positions, and the texture matches a bird, it confidently identifies the animal. This method allows the computer to ignore distracting changes in color or style and focus on the geometry that defines the object.

The researchers found that the system's ability to generalize depended heavily on how well it could locate these parts. When the system correctly identified the key regions, such as the neck or the feet of a bird, it made accurate predictions. However, when the system failed to anchor on these specific areas, often collapsing its attention onto the body of the bird instead, it was more likely to make a mistake. This highlights that the success of the approach relies on the quality of the initial part detection. By treating the image as a composition of localized parts and their relationships, the researchers created a model that is more robust to the unpredictable changes found in the real world. The results indicate that understanding the structural skeleton of an object is a more reliable path to recognition than relying solely on the surface details that can vary so wildly from one domain to another.

Technical Summary: CRISP – Compositional Relations as Invariant Structural Priors for Domain Generalization

Problem Statement

Domain Generalization (DG) aims to train classifiers that remain accurate on unseen target domains, despite shifts in low-level attributes such as color, texture, lighting, or visual style. While existing DG methods often focus on aligning feature distributions, perturbing samples, or regularizing optimization, they largely rely on deep backbone representations to implicitly capture layout, part structure, and spatial relations. The authors argue that leaving this shared compositional structure implicit causes models to overlook a reliable source of invariance. Consequently, classifiers may fail when domain shifts alter local appearances but preserve the underlying structural relationships among components (e.g., a bird's beak remains near its eyes regardless of whether the image is a photograph or a cartoon).

Methodology: CRISP

The authors propose Compositional Relational Invariance from Spatial Primitives (CRISP), an end-to-end framework that explicitly factors visual recognition into learned visual primitives and their relational composition.

1. Visual Primitives and Descriptors

CRISP assumes a set of $K$ learned visual primitives. For each primitive $p_k$ , the model learns a descriptor $z_k(X)$ comprising:

Spatial Location ( $c_k$ ): Differentiable coordinates.
Presence Score ( $\sigma_k$ ): Confidence of the primitive's existence.
Spatial Extent ( $\delta_k$ ): The size of the primitive region.
Appearance Descriptor ( $\ell_k$ ): An attention-pooled feature vector from the primitive's region.

2. Differentiable Spatial Predicates

To model structural invariance, CRISP employs a vocabulary of soft, differentiable predicates that operate on primitive descriptors:

Unary Appearance Predicates: A shared vocabulary of learned tests applied to appearance descriptors to identify attributes (e.g., "curved," "dashed").
Binary Spatial Predicates: Capture pairwise relations such as relative position (above, left), alignment (horizontal/vertical), proximity, and containment.
Ternary Spatial Predicates: Approximate higher-order geometric cues, such as triangular arrangements ( $R_{tri}$ ) and turning angles along chains of primitives ( $R_{turn}$ ).

3. Network Architecture

The framework consists of three jointly optimized components:

Visual Backbone: Extracts general features (e.g., ResNet-50 or ViT-B).
Concept Bottleneck Layer: Projects backbone features into $K$ primitive heatmaps. Using differentiable heatmap-to-coordinate transforms, it extracts the spatial coordinates, presence scores, and appearance descriptors for each primitive.
Structural Scoring Layer:
- Structural Branch: Evaluates class-specific sparse compositions of the learned predicates. It enumerates valid assignments of primitives to predicate arguments and computes a weighted score based on class-specific structural weights ( $\lambda_c$ ).
- Appearance Branch: A parallel Region-of-Interest (ROI) head aggregates direct class evidence from the primitive regions.
- Final Prediction: Class probabilities are derived from the joint evidence of structural compositions and localized primitive appearance.

The model is trained end-to-end using a loss function combining cross-entropy, regularization for stable predicate learning, and a domain-adversarial term to encourage domain-invariant backbone features.

Key Contributions

Structure-Aware Framework: A novel approach to DG that treats visual categories as compositions of learned primitives and spatial relations, explicitly modeling structural invariance rather than relying on implicit backbone representations.
End-to-End Differentiable Architecture: A system that learns primitive detectors, structural predicate parameters, and class-specific compositions simultaneously from image-level supervision, without requiring manual part annotations.
State-of-the-Art Performance: The framework achieves new state-of-the-art results on standard DG benchmarks, demonstrating that explicit relational composition complements local appearance evidence.

Experimental Results

The authors evaluated CRISP on the DomainBed suite (covering shifts in style, provenance, and camera-trap location) and CUB-DG (a fine-grained bird recognition benchmark sensitive to compositional shifts).

DomainBed:
- With ResNet-50, CRISP achieved 70.1% mean accuracy, surpassing the previous state-of-the-art (ERM++) by 1.2 percentage points.
- With ViT-B/14, CRISP achieved 79.0% mean accuracy, outperforming ERM++ by 0.4 percentage points.
CUB-DG:
- CRISP achieved a 69.0% mean accuracy with ResNet-50, a significant improvement of 7.9 percentage points over ERM++. This highlights the method's effectiveness in scenarios where fine-grained recognition depends on part-level attribute combinations.
Ablation Studies:
- Removing the structural scoring layer (relying only on appearance) or the ROI head consistently degraded performance.
- Including both binary and ternary predicates yielded the best results, confirming the value of modeling higher-order geometric cues.
- Visualization showed that learned primitives consistently localized class-relevant regions (e.g., wings, necks) across domains, even when textures changed.

Significance and Claims

The paper claims that CRISP addresses a critical limitation in current DG methods: the under-specification of structural composition. By explicitly representing visual categories through spatial relations among learned primitives, the model leverages a source of invariance that persists even when low-level appearance features shift.

The authors position CRISP not merely as a performance booster but as a structural inductive bias that aligns with long-standing theories of recognition (e.g., Biederman, 1987). The results suggest that while strong backbones and feature alignment are effective, explicitly modeling compositional relations provides a more robust mechanism for generalization, particularly in fine-grained tasks where the arrangement of parts is diagnostic. The paper concludes that combining explicit relational composition with local appearance evidence is a promising direction for improving domain generalization.