⚡ electrical engineering

Learning QoE from Packet-Level Measurements in Encrypted Video Conferencing Traffic

This paper proposes a lightweight, CNN-based framework that accurately predicts video conferencing quality of experience (QoE) metrics like BRISQUE and MOS using only encrypted packet size data, enabling ISPs to assess performance without accessing content.

Original authors: Michael Sidorov, Ofer Hadar

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Michael Sidorov, Ofer Hadar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge the quality of a video call, but the conversation is happening inside a locked, soundproof glass box. You can't see the faces or hear the voices (because the data is encrypted), and you certainly can't ask the people inside how they feel. All you have is a window where you can see the size of the boxes being thrown back and forth and how fast they are arriving.

This is the challenge faced by Internet Service Providers (ISPs) today. They want to know if a user is having a "good" or "bad" video call experience (called Quality of Experience, or QoE), but because of modern security encryption, they can't look inside the data packets. They only see the "envelopes" (packet sizes) and the timing.

Here is how the authors of this paper solved that puzzle, explained simply:

1. The Problem: The "Black Box" Mystery

In the old days, ISPs could peek inside the data packets (like opening a letter) to see if the video was blurry or frozen. But now, almost everything is encrypted. It's like trying to guess the quality of a movie just by watching the delivery truck's tire tracks.

The Goal: Predict how happy a user is with their video call (their QoE) using only the size and timing of the encrypted data packets.
The Difficulty: The relationship between "packet size" and "user happiness" is messy and non-linear. It's not a simple math formula; it's a complex pattern.

2. The Solution: The "qCNN" Detective

The authors built a new tool called qCNN (a lightweight Convolutional Neural Network). Think of this as a super-smart detective that looks for patterns in the "tire tracks" to guess the movie quality.

Here is how the detective works, step-by-step:

Step A: Turning a Line into a Picture (The EMBD Module)

Usually, data comes in as a long, flat line of numbers (packet sizes over time). The authors realized that looking at a long line is hard for a computer to find complex patterns in.

The Analogy: Imagine you have a long strip of film. It's hard to see the whole story at once. So, they took that strip and folded it into a square grid, turning the line of numbers into a little image.
Why? Computers are very good at recognizing shapes in images (like recognizing a cat in a photo). By turning the data stream into a "picture," the computer can spot hidden patterns that look like shapes, rather than just a list of numbers.

Step B: The "ResNet" Eye (The HEAD Module)

Once the data is a "picture," they feed it into a pre-trained eye called ResNet-18.

The Analogy: This is like hiring a detective who has already spent years studying thousands of photos of cats, dogs, and cars. Even though the "picture" here isn't a real photo of a cat, the detective's brain is already trained to spot edges, curves, and textures.
The Trick: They didn't train the detective from scratch (which takes a long time and lots of data). They used a "pre-trained" brain and just taught it a few new tricks specific to video calls. This made the system fast and accurate even with limited data.

Step C: The Prediction (The PRED Module)

Finally, the detective looks at the patterns it found in the "image" and gives a score.

The Score: It predicts two things:
1. BRISQUE: A computer score for how "blurry" or "distorted" the image looks (0 to 100, where lower is better).
2. MOS: A "Mean Opinion Score," which is basically a guess at what a human would rate the call (1 to 5 stars).

3. The Secret Sauce: "Saw-LR" Learning

To teach this detective, the authors used a special training schedule called Saw-LR.

The Analogy: Imagine teaching a student to run. If you tell them to run at the same speed forever, they might get stuck in a rut. Instead, the authors made the student run fast, then slow down, then speed up again, then slow down even more.
The "Saw" Shape: The learning rate goes up and down like the teeth of a saw. This helps the computer "jump" out of bad spots (local minimums) and find the best possible solution, rather than getting stuck halfway.

4. What They Tested

They tested their system on real video calls from WhatsApp and Zoom.

The Setup: They simulated bad internet (slow speed, lost packets) and recorded the traffic.
The Result: Their "qCNN" detective was better than all the other methods they compared it against.
- It beat traditional math models.
- It beat other complex AI models (like LSTM and TCN).
- It worked surprisingly well even with a small dataset (Zoom had only 500 samples), proving it doesn't need a massive library of data to be smart.

5. Key Takeaways

No Peeking Allowed: You don't need to break encryption to know if a video call is bad. You just need to look at the "envelope sizes" and "delivery times."
Shape Matters: Turning a list of numbers into a 2D "image" helps AI find patterns much better.
Simple is Powerful: You don't need a supercomputer. This system is lightweight, easy to build, and runs efficiently.
The "Stationary" Clue: They found that when the network traffic is "boring" and steady (stationary), the call is usually good. When the traffic is "jumpy" and chaotic (non-stationary), the call quality is usually dropping.

Summary

The authors built a smart, lightweight AI that acts like a traffic pattern detective. By turning encrypted data streams into little images and using a pre-trained "eye" to spot shapes, it can accurately guess how happy a user is with their video call, all without ever seeing the video or hearing the audio. It's a new way to keep the internet running smoothly, even when the content is locked away.

Technical Summary: Learning QoE from Packet-Level Measurements in Encrypted Video Conferencing Traffic

Problem Statement

The proliferation of video conferencing applications (VCAs) like Zoom and WhatsApp, accelerated by the post-pandemic shift to remote work, has heightened the need for robust Quality of Experience (QoE) monitoring. While Content Providers (CPs) can assess quality by comparing transmitted and received data, Internet Service Providers (ISPs) face significant challenges due to the widespread adoption of end-to-end encryption. This encryption renders traditional Deep Packet Inspection (DPI) ineffective, as ISPs cannot access payload content to evaluate quality.

Consequently, ISPs must rely on passive measurements of unencrypted traffic characteristics (Quality of Service, or QoS) to infer QoE. However, the relationship between QoS metrics (e.g., packet loss, jitter) and user-perceived QoE is highly non-linear and complex. Existing machine learning (ML) and deep learning (DL) approaches often suffer from overfitting on small datasets, rely on application-level features unavailable in encrypted traffic, or require computationally expensive architectures unsuitable for real-time deployment. Furthermore, many existing models depend on handcrafted statistical features or specific application indicators (e.g., rebuffering events) that are inaccessible to network operators.

Methodology

The authors propose qCNN, a lightweight, simple, yet effective framework for predicting QoE metrics (specifically BRISQUE and Mean Opinion Score, or MOS) using only low-level packet measurements from encrypted traffic. The framework operates without application-level information or payload inspection.

1. Input Features

The model utilizes two primary low-level traffic features extracted from encrypted streams:

Packet Size (PCKT): The size of individual packets.
Packet Inter-Arrival Time (PIAT): The time interval between consecutive packets.
These features are aggregated over 0.5-second intervals. The authors hypothesize that these measurements reflect the operational state of the communication channel (e.g., compression levels, congestion, reconnection events) and serve as indirect indicators of perceptual quality.

2. Architecture: qCNN

The architecture consists of three distinct modules:

Embedding (EMBD) Module: A novel component that transforms the one-dimensional sequence of traffic measurements into a two-dimensional representation. Unlike previous approaches that use fixed row-stacking, this module employs a learnable $1 \times 1$ convolutional layer to perform an "inflation" operation. This creates an $M \times M$ tensor, allowing the network to learn weighted combinations and interactions between input features while preserving structural information.
Head (HEAD) Module: A Convolutional Neural Network (CNN) feature extractor based on the ResNet-18 architecture. This module processes the 2D embedded representation to extract locally salient features. The authors utilize transfer learning, initializing the HEAD module with weights pre-trained on ImageNet-1K. To balance performance and computational cost, the first two residual blocks are frozen, while the remaining layers are fine-tuned.
Predictor (PRED) Module: A fully connected layer that maps the high-level features extracted by the HEAD module to a scalar output representing the predicted QoE metric.

3. Training and Optimization Strategies

Data Augmentation: To improve robustness and generalization, the training pipeline applies additive Gaussian noise and random permutation of elements within the input vector with a probability of 0.2.
Learning Rate Schedule: The authors introduce a Saw-LR schedule, a cyclical learning rate strategy inspired by Smith's work. This schedule gradually decreases the learning rate with periodic spikes to help the optimizer escape local minima, demonstrating superior performance compared to fixed learning rates.
Regularization: Dropout is employed with a low probability ( $p=0.01$ ) after the initial convergence phase to prevent overfitting.

4. Datasets

The study utilizes two custom datasets collected under controlled experimental conditions:

WhatsApp (WA) Dataset: 107 video sessions (4 minutes each) labeled with BRISQUE (an objective, no-reference image quality metric). The dataset contains 10,607 samples with varying bandwidth and packet loss conditions.
Zoom (ZO) Dataset: 30 video clips (6 seconds each) labeled with Mean Opinion Score (MOS), a subjective quality metric derived from human ratings. This dataset contains 500 samples.

Key Contributions

Novel Framework: Introduction of qCNN, a CNN-based framework for QoE prediction from passive, low-level traffic measurements in encrypted channels, eliminating the need for application-level information.
Learnable 1D-to-2D Transformation: Development of the EMBD module, which learns to transform traffic sequences into a 2D feature space, enhancing the CNN's ability to capture complex dependencies compared to manual or static transformations.
Traffic Stationarity Analysis: Investigation of the stationarity properties of low-level traffic. The study demonstrates that non-stationary signals (indicating network changes or degradation) are often more informative for QoE prediction than stationary signals.
Open Datasets: Public release of two custom QoE datasets (WhatsApp and Zoom), with the Zoom dataset being a rare resource annotated with subjective MOS labels for video conferencing.
Performance Benchmarking: Demonstration that the proposed framework achieves superior prediction performance compared to classical ML models (SVM, Random Forest, XGBoost) and other DL architectures (LSTM, TCN, CNNQoE) on both datasets.

Experimental Results

The qCNN model was evaluated using 5-fold cross-validation against a suite of baseline models.

WhatsApp Dataset (BRISQUE): qCNN achieved the best performance across all metrics, with a Mean Absolute Percentage Error (MAPE) of 7.16%, a Root Mean Squared Error (RMSE) of 6.70, and a Linear Correlation Coefficient (LCC) of 0.88. It significantly outperformed the next best model (CatBoost) and vastly outperformed other DL models like TCN and LSTM-QoE, which showed negative or near-zero correlation in some metrics.
Zoom Dataset (MOS): Despite the small dataset size (500 samples), qCNN achieved the lowest MAPE (9.04%) and Mean Absolute Error (MAE, 0.2764) among all tested models. It demonstrated robustness in limited-data conditions, though the authors note the statistical limitations of drawing strong conclusions from such a small sample size.

Ablation Studies confirmed that the CNN HEAD module and the Saw-LR learning rate schedule were the most critical components for performance. Removing the CNN layer resulted in a catastrophic drop in performance, and using fixed learning rates yielded inferior results compared to the proposed Saw-LR schedule.

Significance and Claims

The paper claims that qCNN provides a simple, easy-to-implement, and computationally efficient solution for QoE prediction in encrypted traffic. By relying solely on packet sizes and inter-arrival times, the framework enables ISPs to monitor user experience without violating privacy or requiring access to decryption keys.

The authors emphasize that the framework's ability to leverage transfer learning (via ImageNet pre-trained weights) and learnable embeddings allows it to generalize effectively even with relatively small datasets. They position the work as a step toward wider accessibility in QoE monitoring, offering an open-source solution that surpasses previous state-of-the-art models in prediction accuracy while maintaining low computational overhead suitable for edge and IoT deployment.

The paper remains modest regarding its scope, acknowledging that the Zoom dataset is small and that future work is required to validate the model on larger, more diverse datasets and across a broader range of video conferencing platforms. It also suggests that exploring deeper ResNet variants and transformer-based architectures represents a promising direction for future research.