⚡ electrical engineering

Efficient Audio-Visual Event Recognition via Knowledge Distillation and Dynamic INT8 Quantization of a Hybrid Cross-Attention Network

This paper proposes an efficient audio-visual event recognition framework that combines a lightweight student model trained via knowledge distillation from a high-capacity teacher and dynamic INT8 quantization to significantly reduce model size and parameters while maintaining competitive accuracy for deployment on resource-constrained edge devices.

Original authors: Parinaz Binandeh Dehaghani, Danilo Pena, A. Pedro Aguiar

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Parinaz Binandeh Dehaghani, Danilo Pena, A. Pedro Aguiar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world not just by looking at it, but by listening to it too. This is the world of Audio-Visual Event Recognition (AVER). Think of it like a detective who solves crimes by looking at a scene and listening to the sounds around it. If a detective only sees a broken window, they might think it was a storm. But if they also hear a baseball bat crack, they know it was a kid playing ball. Computers are getting really good at this "detective work" using special brain-like structures called Transformers. These are like super-smart libraries that can read a whole book (or watch a whole video) at once to find connections between words or sounds.

However, there's a catch. These super-smart libraries are huge, heavy, and hungry for energy. They are like a massive, high-end supercomputer that needs a whole room to cool down. If you want to put this detective on a small robot, a smartwatch, or a drone that flies around a city, that supercomputer is too big and eats too much battery. The big question for scientists is: How do we shrink this giant brain down to fit in a pocket without making it forget how to be smart? This is the puzzle a team of researchers from the University of Porto and ResoSight decided to solve.

The Big Idea: A Master Chef and a Sous-Chef

The researchers came up with a clever three-step plan to shrink their giant audio-visual detective without losing its sharpness. They treated the problem like training a new employee in a busy kitchen.

Step 1: The Master Chef (The Teacher)
First, they built a "Teacher" model. This is the giant, super-accurate detective. It uses two powerful tools: one to understand video (called VideoMAE) and one to understand sound (called the Audio Spectrogram Transformer). These tools are like two expert chefs who have already learned everything about cooking. The Teacher combines their knowledge using a special "Cross-Attention" mechanism. Imagine this as the two chefs constantly whispering to each other, saying, "Hey, look at that sound!" or "Did you see that movement?" This helps them understand the full story. But, this Teacher is too heavy to carry around.

Step 2: The Lightweight Apprentice (The Student)
Next, they built a "Student" model. This is the lightweight version, designed to fit on a small device. Instead of building a new brain from scratch, they took the Teacher's brain and made it smaller. They didn't change the way the brain works (the architecture); they just made the parts smaller.

They reduced the "hidden dimension" (how much information the brain holds at once) from 512 to 256.
They cut the number of "attention heads" (the number of things the brain can focus on at once) from eight down to four.
They shrunk the "feed-forward network" (the part that processes the information) from 1024 to 512.

It's like taking a massive library and removing half the shelves and books, but keeping the same layout so the librarian still knows where everything goes.

Step 3: The Secret Tutoring (Knowledge Distillation)
Here is the magic trick. You can't just shrink a brain and expect it to work; it would be like giving a student a smaller backpack and expecting them to know the same amount of math. So, the researchers used Knowledge Distillation.
Imagine the Master Chef (Teacher) cooking a complex dish. Instead of just giving the Apprentice (Student) the recipe, the Chef lets the Apprentice taste the dish and explains why it tastes good. The Apprentice learns not just the final answer (what event happened), but the "soft" feelings and relationships between different sounds and sights. The Student learns to mimic the Teacher's thinking process. This way, even though the Student is smaller, it learns to think like the giant.

Step 4: The Digital Compression (Dynamic INT8 Quantization)
Finally, even after shrinking the brain, the file size was still a bit big for tiny devices. So, they applied Dynamic INT8 Quantization.
Think of the model's numbers as measurements. Normally, the computer uses very precise measurements (like 32-bit floating-point numbers), which are like measuring a cake with a microscope. It's super accurate but takes up a lot of space. The researchers switched to measuring with a standard ruler (8-bit integers). They didn't need to re-teach the student; they just changed how the numbers were stored. It's like taking a high-definition photo and compressing it into a smaller JPEG file. The image still looks great, but the file is tiny.

What They Found

The results were impressive, like finding a way to fit a full-sized car into a garage without crushing it.

The Shrinkage: The student model ended up with 59.06% fewer trainable parameters than the teacher. In plain English, they cut the size of the "brain" by more than half.
The Smarts: Despite being half the size, the student didn't lose much smarts. Its accuracy dropped by only 2.14% compared to the giant Teacher. It went from getting 84.19% of events right to 82.05%. That's a very small price to pay for making the model so much smaller.
The Final Compression: After the final "ruler" compression (INT8 quantization), the model size dropped from 10.71 MB to just 2.04 MB. That is a massive reduction, making it small enough to fit on many resource-constrained devices.
The Trade-off: The final, super-compressed model still got 81.20% of events right. The researchers noted that while the model became incredibly small, the speed on a standard computer processor didn't get faster (it was actually slightly slower due to the overhead of the new compression method), but the huge savings in memory make it perfect for devices that are running out of space.

Why This Matters

The paper suggests that you don't have to choose between a smart AI and a small AI. By combining architectural shrinking, smart tutoring (distillation), and clever number storage (quantization), they created a framework that keeps the "detective" sharp while making it light enough to carry.

They tested this on the AVE dataset, a collection of over 4,000 videos with sounds, and the method held up well. The researchers are confident that this approach works well for edge AI—those smart devices that live on the "edge" of the internet, like your phone, your car, or a robot, where battery and memory are precious. They didn't claim to have solved every problem in the world, but they showed a very promising path for making powerful audio-visual AI practical for everyday gadgets.

Technical Summary: Efficient Audio-Visual Event Recognition via Knowledge Distillation and Dynamic INT8 Quantization

Problem Statement
Audio-Visual Event Recognition (AVER) has seen significant performance gains through transformer-based multimodal architectures, such as those utilizing the Audio Spectrogram Transformer (AST) and VideoMAE. However, these high-capacity models suffer from high computational complexity, large memory footprints, and expensive attention operations, particularly within hybrid cross-attention fusion networks. These factors hinder their deployment on resource-constrained edge devices (e.g., autonomous robots, wearable systems, and intelligent transportation platforms). While general model compression techniques exist, they are often applied as post-processing steps to generic vision or language models without accounting for the specific architectural characteristics of multimodal fusion networks, potentially disrupting essential cross-modal interaction mechanisms.

Methodology
The authors propose an architecture-aware compression framework designed specifically for hybrid cross-attention-based AVER systems. The framework operates on the trainable multimodal fusion and classification network while preserving the pretrained VideoMAE and AST backbone encoders (used only for offline feature extraction). The approach consists of four integrated stages:

Teacher Architecture: A high-capacity teacher model is trained using supervised learning. It integrates VideoMAE for visual representation, AST for audio feature extraction, and a "Stable Hybrid Cross-Attention" fusion network. This fusion module employs bidirectional cross-attention with residual scaling initialized near zero to ensure training stability while learning cross-modal interactions.
Student Architecture Construction: A lightweight student model is derived by systematically reducing the complexity of the transformer fusion network while maintaining the original topology. Specific architectural reductions include:
- Hidden embedding dimension: 512 $\to$ 256.
- Multi-head attention heads: 8 $\to$ 4.
- Feed-forward network dimension: 1024 $\to$ 512.
Knowledge Distillation: The student model is trained using a combination of standard cross-entropy loss and a distillation loss derived from the teacher's soft predictions. The teacher parameters are frozen during this process. A temperature parameter ( $T=4$ ) and weighting coefficient ( $\alpha=0.8$ ) are used to balance hard-label and soft-label supervision, allowing the student to capture inter-class relationships.
Dynamic INT8 Quantization: Following training, dynamic INT8 post-training quantization is applied to the student model. This converts linear layers from 32-bit floating-point to 8-bit integer precision without requiring retraining. Weights are stored in INT8, while activations are dynamically quantized based on runtime distributions.

Key Contributions

Architecture-Aware Framework: The paper introduces a compression framework tailored to hybrid cross-attention AVER networks, preserving the multimodal reasoning topology while reducing the complexity of the trainable fusion module.
Lightweight Student Design: A specific student architecture is developed that reduces hidden dimensions, attention heads, and feed-forward capacities while retaining the original fusion strategy.
Unified Compression Pipeline: The integration of architectural compression, response-based knowledge distillation, and dynamic INT8 quantization into a single pipeline to transfer knowledge and reduce storage without additional retraining.
Empirical Validation: Extensive experiments on the AVE benchmark demonstrate the framework's ability to balance accuracy and efficiency.

Results
Experiments conducted on the AVE dataset (4,143 video clips, 28 categories) yielded the following results comparing the Teacher, the Compressed Student (KD), and the Quantized Student (INT8):

Parameter Reduction: The student model reduced trainable parameters from 6.86 million to 2.81 million, a 59.06% reduction.
Accuracy Trade-off: The compressed student achieved a test accuracy of 82.05%, representing a 2.14% decrease compared to the teacher's 84.19%. The quantized INT8 model achieved 81.20% (a 2.99% drop from the teacher).
Model Size Reduction:
- The student model size decreased from 26.16 MB (Teacher) to 10.71 MB.
- Dynamic INT8 quantization further reduced the model size to 2.04 MB (an 80.9% reduction from the teacher and 80.9% reduction from the unquantized student).
- Checkpoint size was reduced from 78.53 MB to 4.25 MB.
Performance Metrics: The compressed models maintained competitive weighted F1-scores (82.13% for KD, 81.06% for INT8) and showed stable convergence during training without significant overfitting.

Significance and Claims
The paper claims that the proposed framework offers an effective trade-off between recognition accuracy and computational efficiency, making it a promising solution for deployment on resource-constrained edge AI platforms. By focusing compression on the trainable fusion network rather than the entire pipeline, the method preserves the representational capability of the pretrained backbones while substantially reducing the memory footprint of the reasoning component. The authors assert that the combination of architectural simplification, knowledge distillation, and dynamic quantization allows for an 80%+ reduction in model size with less than a 3% accuracy drop, validating the approach for practical edge deployment. The paper notes that while inference latency on CPU was slightly higher for the quantized model due to overhead, the storage savings are the primary benefit for memory-constrained environments. Future work is suggested to explore structured pruning, quantization-aware training, and evaluation on larger-scale benchmarks.

The Big Idea: A Master Chef and a Sous-Chef

What They Found

Why This Matters

Technical Summary: Efficient Audio-Visual Event Recognition via Knowledge Distillation and Dynamic INT8 Quantization

More like this