🤖 AI

Progressive Multimodal Interaction Network for Reliable Quantification of Fish Feeding Intensity in Aquaculture

This paper proposes a Progressive Multimodal Interaction Network (PMIN) that integrates image, audio, and water-wave data through unified feature extraction, cross-modal attention mechanisms, and adaptive evidence reasoning to achieve highly accurate and robust quantification of fish feeding intensity for precision aquaculture.

Original authors: Shulong Zhang, Mingyuan Yao, Jiayin Zhao, Daoliang Li, Yingyi Chen, Haihua Wang

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shulong Zhang, Mingyuan Yao, Jiayin Zhao, Daoliang Li, Yingyi Chen, Haihua Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess how hungry a school of fish is just by watching them eat. In the old days, a farmer would stand by the pond, squint at the water, and guess, "Hmm, they seem pretty hungry, let's throw in more food." But humans get tired, they get distracted, and they might be wrong. If you feed too much, the food rots and pollutes the water; if you feed too little, the fish starve.

This paper introduces a "super-smart assistant" for fish farmers called PMIN (Progressive Multimodal Interaction Network). Think of it as a three-sensory detective that solves the mystery of "How hungry are the fish?" by combining three different clues at once.

Here is how it works, broken down into simple concepts:

1. The Three Detectives (The Data)

Instead of just looking at the fish, this system uses three different "senses" to get the full picture:

The Eyes (Video): A camera watches the fish. It sees them swarming together and splashing water. But, if the sun is too bright or the water is murky, the eyes get confused.
The Ears (Audio): A microphone underwater listens to the "crunch, crunch, crunch" of fish eating. But, if the wind is blowing or the water is flowing loudly, the ears get distracted.
The Skin (Water Waves): A floating sensor feels the ripples and vibrations on the water's surface. But, a single fish swimming might look like a hungry school, or a big splash might just be a wave.

The Problem: If you ask just the "Eyes" or just the "Ears," they often disagree. The eyes might say "They are eating!" while the ears say "It's just noise." This is called a conflict.

2. The Solution: A Team Huddle (The Network)

The authors built a system called PMIN that acts like a smart team leader managing these three detectives. It has three special tricks:

Trick A: The Universal Translator (Unified Feature Extraction)

Imagine the "Eyes" speak French, the "Ears" speak Spanish, and the "Skin" speaks German. If they try to talk to each other, it's a mess.

What PMIN does: It forces all three to translate their observations into the same language (a "structurally consistent feature space") before they start talking. Now, the camera, the microphone, and the wave sensor are all on the same page, making it easier to compare their notes.

Trick B: The "Captain and Crew" System (Progressive Interaction)

In a real feeding frenzy, one sense is usually the most reliable (usually the Eyes/Video). The other two senses are the "Crew."

How it works: The system picks the "Captain" (the most reliable data, usually the video). Then, it asks the "Crew" (audio and waves), "Hey, does your data support what the Captain sees?"
The Magic: If the Captain sees a splash, but the Crew hears silence, the system doesn't just ignore the Crew. It uses a special "recalibration" tool to ask, "Wait, is the Captain wrong, or is the Crew just missing something?" It gently nudges the Captain's opinion with the Crew's info, creating a stronger, more accurate conclusion. It's like a coach saying, "The striker is open, but the defense is tight; let's adjust the play."

Trick C: The Jury Room (Adaptive Evidence Reasoning)

After the team discusses, they have to vote on the final answer: Strong Hunger, Weak Hunger, or No Hunger.

The Problem: Sometimes the "Eyes" are 90% sure it's "Strong Hunger," but the "Ears" are only 40% sure. A simple average might get it wrong.
The Solution: PMIN acts like a wise judge in a jury room. It doesn't just count votes; it asks, "How much do we trust this witness?"
- If the camera is blurry, the judge lowers its trust score.
- If the microphone is clear, the judge boosts its trust score.
- It weighs the confidence and reliability of each piece of evidence to make the final decision. This prevents one noisy sensor from ruining the whole group's judgment.

3. The Results: A Star Student

The researchers tested this system on a dataset of nearly 7,000 fish feeding sessions.

The Score: The system got 96.76% accuracy. That's like getting an A+ on a very hard test.
Comparison: It beat other methods that only used one sense (like just a camera) or older ways of combining senses.
Real World: When they tested it in a real fish farm (not just a computer simulation), it worked almost perfectly, tracking the fish's hunger levels smoothly over time.

Why This Matters

Think of this system as the difference between a guessing game and precision engineering.

Old Way: "I think they are hungry, throw in a bucket." (Result: Some fish eat, some starve, some water gets dirty).
New Way (PMIN): "The eyes see a swarm, the ears hear crunching, and the waves are strong. Confidence is 96%. Throw in exactly 1.5kg of food." (Result: Happy fish, clean water, saved money).

In short, this paper teaches computers how to be better observers by teaching them to listen to all their senses, trust the right ones, and resolve arguments before making a decision. It's a big step toward fully automated, smart fish farming.

1. Problem Statement

Accurate quantification of fish feeding intensity is critical for precision feeding in aquaculture to optimize feed utilization and prevent waste or disease. However, existing methods face significant challenges:

Single-Modality Limitations: Visual methods are susceptible to water turbidity and lighting; acoustic methods suffer from environmental noise; sensor-based methods are often invasive or lack group-level representation.
Multimodal Deficiencies: While multimodal fusion is promising, existing approaches often:
- Ignore inconsistencies in response intensity and decision conflicts between different modalities (e.g., water ripples may be strong while visual feeding cues are weak).
- Rely on simple fusion strategies (concatenation, static weighting) that treat modalities as independent units rather than modeling their complex, differentiated contributions.
- Lack explicit modeling of confidence, reliability, and conflict at the decision level, leading to unstable quantification results.

2. Methodology: Progressive Multimodal Interaction Network (PMIN)

The authors propose PMIN, a three-stage framework designed to integrate image, audio, and water-wave data for robust feeding intensity quantification.

A. Unified Feature Extraction Framework

Backbone: Utilizes UniRepLKNet (Universal Perception Large-Kernel ConvNet) as a shared backbone for all modalities.
Strategy: Instead of using heterogeneous encoders for each modality, inputs (RGB images, Mel spectrograms of audio, and time-series sensor data) are transformed into a structurally consistent C×H×W tensor format.
Benefit: This reduces representational discrepancies across modalities and establishes a unified feature space, facilitating more effective cross-modal interaction.

B. Auxiliary-Modality Reinforcement Primary-Modality Mechanism (ARPM)

To address the varying discriminative power of different modalities under specific conditions, PMIN employs a progressive interaction mechanism:

Primary/Auxiliary Roles: In each branch, one modality acts as the "primary" (e.g., Image), while others act as "auxiliary."
Two-Stage Interaction:
1. Shallow Interaction (DAIM-1): Uses Multi-head Self-Attention (MHSA) to model internal correlations within auxiliary features, followed by Multi-head Cross-Attention (MHCA) to inject discriminative auxiliary information into the primary modality.
2. Deep Interaction (DAIM-2): A parallel cross-modal attention mechanism that fuses shallow enhanced features bidirectionally to capture deep complementary semantics.
Channel Aware Recalibration (CARM): Integrated at both stages to adaptively recalibrate channel weights, mitigating representational discrepancies and solidifying modality weights.

C. Decision Fusion based on Adaptive Evidence Reasoning (AER)

To resolve decision conflicts and model uncertainty:

Evidence Modeling: Converts classification outputs from each modality branch into Basic Probability Assignments (BPA).
Adaptive Optimization: Unlike traditional methods with fixed weights, AER treats evidence weight ( $w$ ) and reliability ( $r$ ) as learnable parameters.
Fusion Rule: Applies an evidential reasoning rule to jointly model confidence, reliability, and conflict. This suppresses interference from conflicting evidence while preserving strong discriminative cues, producing a robust global decision.

3. Key Contributions

Dataset Construction: Created a multimodal dataset (7,089 samples) synchronizing image, audio, and water-wave data from rainbow trout, covering direct behavioral and environmental response representations.
Unified Feature Extraction: Proposed a framework using UniRepLKNet to encode heterogeneous data into a consistent feature space, reducing alignment complexity.
Progressive Interaction Mechanism (ARPM): Designed a novel mechanism that progressively enhances primary modality representations through channel-aware recalibration and dual-stage attention, effectively modeling differentiated modality responses.
Adaptive Decision Fusion (AER): Introduced a decision-level fusion strategy that dynamically learns evidence reliability and conflict, significantly improving stability over static fusion methods.
Comprehensive Validation: Demonstrated superior performance through extensive ablation studies, comparisons with homogeneous/heterogeneous models, and real-world application testing.

4. Experimental Results

Experiments were conducted on a dataset of 7,089 samples (8:1:1 train/val/test split).

Overall Performance: PMIN achieved 96.76% Accuracy, 96.80% Precision, 96.78% Recall, and 96.79% F1-Score.
Comparison with Unimodal/Bimodal:
- Unimodal Image: 82.39% Accuracy.
- Unimodal Audio: 59.86% Accuracy.
- Unimodal Water-wave: 66.34% Accuracy.
- PMIN (Trimodal) outperformed all unimodal and bimodal combinations, confirming the necessity of multimodal collaboration.
Comparison with State-of-the-Art:
- Outperformed homogeneous models (e.g., OverLoCK-XT: 94.51%) and various heterogeneous configurations (which showed high instability, dropping to 76.34% in some cases).
- Efficiency: PMIN achieved the highest accuracy with only 14.39M parameters and 1.88 GFLOPs, making it suitable for edge deployment.
Ablation Studies:
- Removing ARPM dropped accuracy by ~7%.
- Removing AER dropped accuracy by ~1.7%.
- The combination of both modules yielded the best results, proving their functional complementarity.
Real-World Application: Tested on an unseen 240-second continuous dataset, achieving 95.82% accuracy with high temporal continuity, validating its suitability for real-time monitoring.

5. Significance

Reliability: By explicitly modeling modality conflicts and confidence, PMIN provides more reliable quantification than methods relying solely on feature concatenation or simple voting.
Practicality: The unified architecture and low computational cost enable deployment in resource-constrained smart aquaculture environments.
Scientific Insight: The study highlights that visual data is the dominant modality for feeding intensity, but audio and water-wave data provide crucial complementary cues for robustness in noisy or turbid conditions.
Impact: The method offers a robust technical foundation for automated feeding systems, directly contributing to reduced feed waste, lower costs, and improved sustainability in global aquaculture.