📊 statistics

Stylistic-STORM (ST-STORM) : Perceiving the Semantic Nature of Appearance

This paper introduces ST-STORM, a hybrid self-supervised learning framework that disentangles content and style into separate latent streams to effectively capture appearance-based semantic signals for tasks like weather analysis and melanoma detection without compromising object recognition performance.

Original authors: Hamed Ouattara, Pierre Duthon, Pascal Houssam Salmane, Frédéric Bernardin, Omar Ait Aider

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hamed Ouattara, Pierre Duthon, Pascal Houssam Salmane, Frédéric Bernardin, Omar Ait Aider

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Blindfolded" AI

Imagine you are teaching a robot to recognize a cat.

The Old Way: You show the robot a cat in bright sun, a cat in the dark, a cat covered in snow, and a cat with a red hat. The robot learns to ignore the sun, the snow, and the hat. It focuses only on the "cat-ness" (ears, whiskers, tail). This is great for finding cats, but it's terrible if you need to know what the cat is wearing or if it's snowing outside.
The Flaw: In many real-world jobs, those "ignored" details are actually the most important clues.
- Driving: If a self-driving car ignores the rain streaks on the windshield, it might not realize the road is slippery.
- Medicine: If a doctor ignores the texture of a skin mole, they might miss a sign of cancer.

Current AI models are like that robot: they are trained to be "invariant" (unaffected by changes). They treat rain, fog, and skin texture as "noise" to be filtered out. But sometimes, the noise is the signal.

The Solution: ST-STORM (The "Bilingual" Brain)

The authors created a new system called ST-STORM. Instead of forcing the AI to ignore appearance, they taught it to speak two languages at once: Content and Style.

Think of ST-STORM as a bilingual brain with two specialized assistants:

1. The "Content" Assistant (The Architect)

Job: This assistant looks at the structure. It asks: "What is this object? Where is it? What shape is it?"
Training: It is trained to be tough. It sees the same car in rain, snow, and sunshine and says, "That's still a car." It learns to ignore the weather to understand the object.
Result: It is excellent at standard tasks like recognizing a cat or a car, just like older AI models.

2. The "Style" Assistant (The Artist)

Job: This assistant looks at the atmosphere. It asks: "Is it raining? Is the skin rough? Is the light hazy?"
Training: This is the magic part. Instead of ignoring the rain, this assistant is trained to study it. It learns to separate the "rain" from the "car."
The Trick: The system uses a technique called Style Transfer (like a digital painter). It takes a photo of a sunny street and "paints" rain over it. Then, it forces the Style Assistant to predict what the rain looks like without looking at the car.
Result: It becomes an expert at detecting fine details like skin texture (for cancer) or weather conditions (for driving).

How It Works: The "Chaos" Kitchen

To teach these two assistants, the researchers set up a chaotic kitchen:

The Ingredients: They take a picture (the source) and a different picture (the style reference).
The Mixer: They use a special tool (a neural network) to mix them. They keep the shape of the source (the car) but swap the texture with the reference (making it look like it's in a snowstorm).
The Test (The "Jeopardy" Game):
- The Content Assistant is tested: "Can you still tell this is a car even though it's covered in fake snow?" (Answer: Yes, because it learned to ignore the snow).
- The Style Assistant is tested: "Can you describe the snow without looking at the car?" (Answer: Yes, because it learned to focus only on the snow).

Why This Matters: Real-World Superpowers

The paper tested this on three difficult tasks, and the results were impressive:

Weather Analysis:
- The Task: Is it raining? Is it foggy? Is the road wet?
- The Result: The Style Assistant was a superstar. It got 97% accuracy. Old models (MoCo, I-JEPA) got stuck around 87-91% because they tried to "ignore" the rain. ST-STORM realized the rain was the answer.
Cancer Detection (Melanoma):
- The Task: Is this skin spot dangerous?
- The Result: Dangerous spots often have weird textures and colors. The Style Assistant spotted these micro-details with 94% accuracy, beating the other models.
Object Recognition (ImageNet):
- The Task: Just identifying objects (cats, dogs, cars).
- The Result: The Content Assistant didn't get confused. It still recognized objects just as well as the best existing models. The system didn't lose its "common sense" while gaining its "artistic eye."

The Secret Sauce: "Predictability"

Why didn't the Style Assistant just memorize random noise?
The researchers used a clever rule called Style-JEPA.

Imagine asking a student to predict the next word in a sentence. If the sentence is random gibberish, they can't predict it. If the sentence follows a pattern, they can.
The system forces the Style Assistant to predict the "style" of an image based on its surroundings. If the style is just random noise, the AI fails. If the style is a real pattern (like "heavy rain" or "smooth skin"), the AI succeeds. This forces the AI to learn meaningful patterns, not just random pixels.

The Bottom Line

ST-STORM is a breakthrough because it stops treating "appearance" (weather, texture, lighting) as a nuisance. Instead, it treats appearance as a second language.

Old AI: "I see a car. I ignore the rain."
ST-STORM: "I see a car (Content), AND I see that it is raining heavily on a slippery road (Style)."

This makes AI much safer for self-driving cars and more accurate for medical doctors, because it finally understands that how things look is just as important as what things are.

1. Problem Statement

Current dominant paradigms in Self-Supervised Learning (SSL), such as MoCo and DINO, aim to learn robust representations by enforcing invariance to image transformations (lighting, color, geometry). While effective for coarse object recognition (e.g., identifying a "cat" regardless of fur color), this approach becomes counterproductive when appearance itself is the discriminative signal.

The Limitation: In fine-grained tasks like weather analysis (rain streaks, fog, snow) or medical imaging (melanoma detection), texture, contrast, and atmospheric scattering are not noise to be filtered out; they are the essential semantic information.
The Gap: Existing predictive methods (like I-JEPA) learn by predicting masked regions but implicitly favor stable, predictable factors, often smoothing out high-frequency appearance details. Reconstruction-based methods (like MAE) tend to memorize contingent, non-transferable pixel details.
Goal: To develop a framework that treats appearance (style) as a distinct, semantic modality that can be disentangled from content, allowing for robust content recognition and precise appearance characterization simultaneously.

2. Methodology: ST-STORM

ST-STORM is a hybrid SSL framework that explicitly disentangles Content (semantic structure) and Style (appearance signatures) into two latent streams.

A. Core Architecture

The model utilizes a dual-branch architecture with a shared generator for data augmentation:

Content Branch: A U-Net encoder designed to extract invariant semantic features. It uses skip connections to preserve geometric structure.
Style Branch: A pyramidal encoder that extracts multi-scale appearance tokens ( $t_1 \dots t_5$ ) and a global token ( $t_G$ ).
Fusion Mechanism (SPADE): The two streams are fused using Spatially-Adaptive Denormalization (SPADE) blocks. The style branch modulates the decoder activations (gain and bias) of the content branch without altering the underlying geometry. This ensures style is treated as a "modulation" rather than a geometric generator.

B. Training Strategy: Pseudo-Domain Cycles

Instead of random augmentations, ST-STORM uses a pseudo-domain partitioning strategy:

The dataset is split into $K$ subsets.
The model performs cyclic style transfer: it takes a source image $x$ (content) and a reference image $y$ (style) from a different subset to generate a stylized image $\tilde{x}$ .
This creates "controlled chaos," forcing the model to learn to separate structure from appearance.

C. Learning Objectives

The framework employs distinct objectives for each branch to enforce the desired properties:

Style Learning (Style-JEPA):
- Goal: Learn a semantic representation of appearance that filters out contingent noise.
- Mechanism: Instead of pixel reconstruction, the model predicts masked style tokens in the latent space based on the visible context.
- Constraint: A predictability loss ( $L_{StyleJEPA}$ ) ensures the model captures stable, repeatable patterns (e.g., "rain" statistics) rather than random pixel noise.
- Adversarial Constraint: A PatchGAN discriminator ensures the stylized image looks realistic in the target domain.
- Token Consistency: The style tokens of the generated image must match the tokens of the reference image.
Content Learning (MoCo with Stylized Positives):
- Goal: Learn a representation invariant to appearance changes.
- Mechanism: Uses a contrastive loss (MoCo-style) where the "positive" pair consists of the original image and its stylized version.
- Effect: This forces the content encoder to ignore the style perturbations introduced by the generator, learning a robust semantic representation.
Guided Reconstruction:
- Ensures that even after heavy style perturbation, the original structure can be recovered, preventing the content information from "leaking" into the style branch.

D. Adaptive Fusion for Downstream Tasks

For tasks requiring both semantics and appearance (e.g., general object recognition), the paper introduces a lightweight adaptive gating mechanism. It projects style and content embeddings into a common space and learns a gate vector to weight their contribution dynamically, allowing the model to decide when to rely on texture vs. structure.

3. Key Contributions

Semantic Disentanglement of Style: Proposes that appearance is a semantic modality, not just noise. It introduces a dedicated branch to learn "Style-JEPA," which predicts stable appearance signatures rather than reconstructing pixels.
Hybrid SSL Framework: Combines contrastive learning (for content invariance) with predictive learning (for style semantics) within a unified architecture.
Pseudo-Domain Cycle Training: A novel training scheme using unpaired style transfer (CycleGAN/CUT principles) to generate diverse appearance variations without requiring paired data.
Adaptive Fusion: A mechanism to combine style and content embeddings, proving that appearance cues can enhance semantic tasks when properly structured.

4. Experimental Results

The model was evaluated on three main tasks:

A. Appearance-Dominated Tasks

Weather Characterization (Weather-MultiTask-Datasets):
- Result: The Style branch achieved F1 = 97.13% (10% labeled data), significantly outperforming MoCo-v3 (91.53%) and I-JEPA (87.27%).
- Insight: The Style branch excelled at fine-grained attributes like precipitation intensity, visibility, and ground conditions, which rely on texture and photometric cues.
Melanoma Detection (ISIC 2024):
- Result: The Style branch achieved F1 = 94.32%, outperforming MoCo-v3 (92.59%) and I-JEPA (92.02%).
- Insight: Critical for medical diagnosis where micro-textures and color irregularities are the primary diagnostic markers.

B. Content-Dominated Tasks

ImageNet-1K Classification:
- Result: The Content branch (ST-C) achieved 75.4% Top-1 accuracy (100% data), outperforming MoCo-v3 (73.8%) with the same ResNet50 backbone.
- Insight: Explicitly modeling style did not degrade semantic performance; in fact, it improved robustness.
Adaptive Fusion: Combining Style and Content (ST-S+C) further improved ImageNet performance to 78.4%, suggesting appearance cues provide complementary information even for object recognition.

C. Generalization

Transfer Learning: ST-STORM showed superior out-of-domain generalization on external weather benchmarks (MWI, WEAPD) compared to baselines, indicating the learned style representations are generic and not overfitted to specific sensors.

D. Ablation Studies

Style-JEPA is Critical: Removing the predictability constraint caused a massive drop in performance (-5.45 F1 points), proving that filtering contingent noise via prediction is the core driver of success.
Frequency Constraints: Explicit frequency losses (FFT, SWD) provided negligible gains, suggesting the latent predictability constraint is sufficient to capture spectral regularities.

5. Significance

Paradigm Shift: Challenges the SSL dogma that "invariance is always good." It demonstrates that for fine-grained analysis, sensitivity to appearance is a necessary semantic capability.
Safety-Critical Applications: Directly addresses the needs of autonomous driving (where rain/snow affect grip and visibility) and medical imaging, where ignoring appearance cues can be dangerous.
Efficiency: Achieves state-of-the-art results on fine-grained tasks with only 10% (or even 1%) labeled data, reducing the annotation burden for critical domains.
Theoretical Insight: Validates that appearance can be modeled as a predictable, structured latent space (Style-JEPA) rather than random noise, bridging the gap between generative modeling and discriminative representation learning.

In conclusion, ST-STORM successfully proves that a self-supervised model can simultaneously learn robust content invariance and discriminative appearance sensitivity by explicitly factorizing these modalities and applying distinct learning objectives to each.