💻 computer science

Cross-modal learning for plankton recognition

This paper proposes a self-supervised cross-modal learning framework that leverages unlabeled plankton image and optical profile data to train multimodal recognition models, achieving high accuracy with minimal labeled data while outperforming image-only baselines.

Original authors: Joona Kareinen, Veikka Immonen, Tuomas Eerola, Lumi Haraguchi, Lasse Lensu, Kaisa Kraft, Sanna Suikkanen, Heikki Kälviäinen

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Joona Kareinen, Veikka Immonen, Tuomas Eerola, Lumi Haraguchi, Lasse Lensu, Kaisa Kraft, Sanna Suikkanen, Heikki Kälviäinen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a marine biologist trying to identify tiny, drifting creatures called plankton in the ocean. These creatures are the "grass of the sea"—they feed the whales, produce our oxygen, and tell us if the ocean is healthy.

The problem? There are billions of them, they look incredibly similar, and they change shape depending on how they are feeling. Traditionally, scientists had to manually look at photos of these plankton and label them one by one. It's like trying to sort a million different types of Lego bricks by looking at a single photo of each one. It's slow, expensive, and boring.

This paper proposes a clever new way to teach computers to do this job, using a method that feels a bit like teaching a child to recognize a dog by showing them a picture and a sound at the same time.

The Two Clues: A Photo and a "Vibe Check"

Most automated systems only look at photos of the plankton. But the special microscope used in this study (called a CytoSense) is like a super-powered detective. As a plankton swims through a laser beam, it leaves two distinct trails of evidence:

The Photo: A bright-field image (a snapshot of what it looks like).
The Profile: A "vibe check" or a unique signature. As the plankton passes through different colored lasers, it scatters light and glows (fluoresces) in specific ways. This creates a graph of six different lines (like a heartbeat monitor) that tells us about the plankton's internal structure and chemistry.

Think of it this way:

The Photo is like seeing a person's face.
The Profile is like hearing their voice or feeling their fingerprint.

Sometimes, two people look identical in a photo, but their voices are totally different. Similarly, two plankton might look the same in a picture, but their "light signature" (profile) reveals they are different species.

The Magic Trick: Learning Without a Teacher

The biggest hurdle in AI is that it usually needs a teacher to say, "This is a Daphnia, and this is a Copepod." But there aren't enough human experts to label millions of images.

The authors used a trick inspired by CLIP (a famous AI that learns to match photos with text descriptions). Instead of matching photos to words, they matched photos to light signatures.

Here is how they taught the AI without labels:

They took a batch of plankton data.
They told the AI: "If this photo and this light signature came from the same plankton, they are a match. If they came from different plankton, they are not."
The AI had to figure out the connection on its own. It learned to build a mental map where the photo of a specific plankton and its unique light signature sat right next to each other, while other species were far away.

It's like teaching a child to recognize a friend by saying, "If you see this face and hear this laugh, it's your friend. If you see this face but hear a different laugh, it's a stranger." The child learns the connection without ever needing the friend's name.

The Result: A Super-Recognizer

Once the AI learned this "multimodal" map, the scientists tested it. They didn't need to show it thousands of labeled examples. They just showed it a tiny "gallery" of known plankton (a few examples of each species) and asked the AI to find the closest match in its mental map.

The findings were impressive:

Better than photos alone: Using both the photo and the light signature was much more accurate than using just the photo. It's like recognizing someone by their face and their voice is harder to fool than just their face.
Works in the wild: The AI trained on messy, real-world ocean data (which is full of different plankton types) was better at identifying new plankton than an AI trained only on perfect, clean lab data.
Less work for humans: Because the AI learned from unlabeled data, scientists don't need to spend years manually labeling every single image.

The Bottom Line

This paper is like giving the ocean a new set of eyes and ears. By teaching computers to look at plankton through two different lenses (vision and light physics) simultaneously, the researchers created a system that is faster, more accurate, and requires far less human effort.

They even released their data and code to the public, hoping that other scientists can use this "multimodal" superpower to keep a closer watch on our planet's most important tiny creatures.

1. Problem Statement

Automated plankton monitoring is critical for understanding aquatic ecosystems and detecting environmental changes. While modern instruments like the CytoSense flow cytometer can generate millions of images and corresponding optical measurement profiles (scatter and fluorescence) per day, current recognition methods face two major bottlenecks:

Label Scarcity: State-of-the-art recognition relies on supervised learning, which requires labor-intensive expert annotation of plankton species.
Modality Underutilization: Existing models typically use only the visual image data. They ignore the rich, complementary optical profile data (e.g., Forward Scatter, Sideward Scatter, and multi-channel fluorescence) that these instruments capture simultaneously.
Generalization Issues: Models trained on specific, labeled datasets often fail to generalize to new environments or species distributions due to domain shifts.

The paper addresses the challenge of building a robust plankton recognition system that leverages unlabeled multimodal data to learn effective representations without requiring extensive manual labeling.

2. Methodology

The authors propose a self-supervised cross-modal learning framework inspired by Contrastive Language-Image Pre-training (CLIP), adapted for plankton data. The pipeline consists of two main stages:

A. Contrastive Image–Profile Pre-training

Instead of aligning images with text (as in CLIP), the model aligns bright-field images with optical profiles of the same plankton particle.

Architecture: Two modality-specific encoders are used:
- Image Encoder: Processes grayscale images (tested: EfficientNet, ConvNeXt, ViT).
- Profile Encoder: Processes sequences of optical signals (6 channels: FSC, SSC, Green, Yellow, Orange, Red) over time/length (tested: 1D CNN, LSTM, Transformer).
Alignment Strategy: The encoders project representations into a shared $d$ -dimensional embedding space.
Loss Functions: The model is trained using two contrastive loss formulations to maximize similarity between matched image-profile pairs and minimize it for mismatched pairs:
1. InfoNCE Loss: Standard symmetric cross-entropy loss.
2. Sigmoid Loss (SigLIP): Independent binary classification for pairs, offering better stability and scalability.
Data: The training utilizes a large volume of unlabeled paired data (image + profile) from the UTO dataset (in-situ Baltic Sea data) and LAB dataset (cultured data).

B. Nearest Neighbor Classification

After pre-training, the learned encoders are frozen. Classification is performed via a k-Nearest Neighbor (k-NN) approach in the embedding space:

Gallery Construction: A small, labeled "gallery" of known plankton species is encoded using the pre-trained encoders.
Inference: A query sample (image, profile, or both) is encoded, and its class is determined by finding the nearest neighbors in the gallery based on cosine distance.
Flexibility: The system supports unimodal (image-only or profile-only) or multimodal (combined) inference. Neighbors from different modalities are aggregated via majority voting.

3. Key Contributions

Multimodal Framework: A novel recognition framework for CytoSense data that effectively fuses bright-field images and optical profiles using contrastive pre-training.
Label-Efficient Learning: Demonstrates that high-accuracy recognition can be achieved using unlabeled multimodal data for pre-training, requiring only a minimal labeled gallery for the final classification step.
New Dataset: Introduction and public release of SYKE-plankton_CytoSense_2025, a multimodal dataset containing three subsets (LAB, SEA, UTO) with over 62,000 samples, including a large unlabeled in-situ dataset.
Cross-Domain Generalization: The method shows superior generalization to unseen environments compared to models trained solely on controlled laboratory data.

4. Experimental Results

The authors evaluated the method on three datasets: LAB (cultured, labeled), SEA (in-situ Baltic Sea, labeled), and UTO (in-situ, unlabeled).

In-Domain Performance (LAB → LAB):
- The best configuration (ViT-Tiny image encoder + CNN profile encoder + InfoNCE loss) achieved 96.01% accuracy when both modalities were used for training and testing.
- Multimodal approaches consistently outperformed unimodal baselines.
- InfoNCE loss outperformed Sigmoid loss in this low-data regime.
- Interestingly, using only optical profiles in the gallery yielded higher accuracy than image-only galleries for certain species, highlighting the discriminative power of optical data.
Cross-Domain Performance (Generalization):
- Models pre-trained on the UTO (unlabeled, in-situ) dataset generalized significantly better to the SEA dataset than models trained on LAB (cultured) data.
- This suggests that raw, diverse in-situ data captures the natural heterogeneity of plankton better than controlled cultures.
- Multimodal vs. Image-Only: The proposed multimodal model consistently outperformed the DINO (image-only self-supervised) baseline, even when evaluated using only image data at inference. This proves that cross-modal pre-training enhances the quality of visual representations.
Ablation Studies:
- Encoder Choice: CNNs generally outperformed Transformers for profile encoding in cross-domain settings, while ViT-Tiny was optimal for images.
- Gallery Size: Accuracy improved steadily as the size of the labeled gallery increased, confirming the robustness of the embedding space.

5. Significance

This work represents a significant shift in plankton monitoring methodology:

Scalability: By utilizing the vast amounts of unlabeled data generated by modern flow cytometers, the need for expensive expert labeling is drastically reduced.
Robustness: The approach leverages the complementary nature of optical and visual data to create representations that are more robust to environmental variations and domain shifts than image-only models.
Practical Application: The ability to classify plankton using a small labeled gallery and a large unlabeled pre-training set makes automated monitoring feasible for long-term, large-scale ecological studies where manual annotation is impossible.

The paper concludes that cross-modal coordination is a powerful strategy for biological recognition tasks, offering a path toward scalable, automated, and accurate plankton monitoring in real-world conditions.