🤖 machine learning

Parameter Efficient Hybrid Transformer (PEHT) for Network Traffic Prediction via Dynamic Urban Congestion Integration

This paper introduces the Parameter-Efficient Hybrid Transformer (PEHT), a framework that integrates urban mobility and congestion data into a Transformer architecture using Low-Rank Adaptation to achieve accurate and resource-efficient network traffic prediction.

Original authors: Abdolazim Rezaei, Mehdi Sookhak, Mahboobeh Haghparast

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Abdolazim Rezaei, Mehdi Sookhak, Mahboobeh Haghparast

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling city like Milan as a giant, living organism. Every time someone makes a phone call, sends a text, or browses the internet, it's like a tiny heartbeat in that organism. City planners and network engineers need to predict where these "heartbeats" will be strongest in the future so they can keep the network from getting a "heart attack" (congestion) or a "faint" (dropped calls).

The problem is, predicting this is incredibly hard. It's not just about looking at the past; it's about understanding how people move, where traffic jams are forming, and how different neighborhoods behave. Traditional methods are like trying to predict the weather by only looking at a single thermometer—they miss the bigger picture.

This paper introduces a new tool called PEHT (Parameter-Efficient Hybrid Transformer). Think of PEHT as a super-smart, highly efficient city traffic controller that uses a special kind of "brain" (an AI model) to forecast network traffic. Here is how it works, broken down into simple concepts:

1. The "Smart Map" (Grid Clustering)

The raw data they used comes from tiny squares on a map of Milan. If you look at just one tiny square, the data is often messy and empty (like trying to hear a whisper in a noisy room).

The Analogy: PEHT doesn't look at individual squares. Instead, it groups neighboring squares together into "Virtual Base Stations." Imagine taking a bunch of small, scattered puzzle pieces and gluing them into a single, clear picture. This creates a smoother, more reliable signal, much like turning a static-filled radio station into a clear broadcast.

2. The "Two-Track Brain" (Hybrid Architecture)

Most AI models try to learn everything at once, which can be overwhelming. PEHT splits the learning into two tracks:

Track A (The Network): It looks at the actual phone and internet data (calls, texts, web browsing).
Track B (The City): It looks at "external" factors, like how many cars are stuck in traffic or how many pedestrians are walking around.
The Analogy: Think of a chef predicting how busy a restaurant will be. Track A is the chef looking at past reservation numbers. Track B is the chef looking out the window to see if a parade is happening nearby or if a storm is rolling in. PEHT combines both views to make a much smarter guess than looking at just one.

3. The "Lightweight Brain" (LoRA)

Usually, AI models that are this smart are huge and heavy, requiring massive computers to run. They are like a supercomputer trying to solve a crossword puzzle.

The Innovation: PEHT uses a trick called LoRA (Low-Rank Adaptation).
The Analogy: Imagine you have a giant encyclopedia (the full AI model) that you need to update with new information. Instead of rewriting the whole book, LoRA is like adding a set of sticky notes with the most important updates. It allows the model to learn effectively while using 90% fewer "brain cells" (parameters). This makes it fast and efficient enough to run on smaller devices, like the NVIDIA Jetson kit mentioned in the paper.

4. The "Time Traveler" (Transformer)

The core of PEHT is a "Transformer," a type of AI famous for understanding long sequences of events.

The Analogy: If you watch a movie, a simple AI might only remember the last 5 seconds. A Transformer remembers the whole plot, understanding how the beginning leads to the end. PEHT uses this to look at hours of past traffic data to predict the future, while strictly ensuring it doesn't "cheat" by peeking at future data it shouldn't know yet.

What Did They Find?

The researchers tested PEHT using real data from Milan and created five different "fake" city scenarios (like a city with only cars, or a city with heavy traffic jams) to see how it held up.

The Result: PEHT was better than all the other top models they tested. It made fewer mistakes (lower error rates) in predicting SMS, Call, and Internet traffic.
The Proof: In their tests, the "Full Model" (using both the smart map, the two-track brain, and the lightweight LoRA trick) consistently outperformed versions that missed any of these parts. It proved that combining city movement data with network data, while keeping the model small and efficient, is the winning strategy.

In short: PEHT is a new, efficient way to predict where your phone signal will be strong or weak by teaching the AI to look at both the phone data and the city's traffic, all while keeping the computer's workload light enough to run easily.

Technical Summary: Parameter Efficient Hybrid Transformer (PEHT) for Network Traffic Prediction

Problem Statement
Accurate network traffic prediction is essential for efficient resource allocation in dynamic urban cellular networks. However, existing models face significant challenges due to the complexity of urban traffic, which is driven by diverse user behaviors, rapidly changing environmental factors, and heterogeneous mobility patterns. Traditional approaches struggle to incorporate external data sources effectively. Specifically, Graph Neural Networks (GNNs) often encounter scalability issues in dense urban environments, while standard Transformer-based models, though effective at capturing long-range dependencies, become computationally and parameter-intensive when handling high-dimensional spatio-temporal data. Furthermore, many existing methods fail to adequately integrate real-time urban mobility and congestion signals without violating temporal causality during inference.

Methodology
The authors propose the Parameter-Efficient Hybrid Transformer (PEHT), a framework designed to integrate urban mobility and congestion information into a Transformer-based architecture while maintaining parameter efficiency. The methodology proceeds through four key stages:

Spatio-Temporal Grid Clustering and Virtual Cell Construction:
To address data sparsity in high-resolution grid squares (using the Telecom Italia Milan dataset), the authors employ a "Grid-to-Virtual-Cell" mapping. Adjacent grid squares are clustered into Virtual Base Stations (VBS) using a greedy algorithm. This algorithm maximizes a utility function $U(g, v)$ that balances spatial proximity, temporal correlation (Pearson coefficient), and load similarity, creating robust signals for prediction.
Multimodal Data Aggregation and Feature Engineering:
The model aggregates diverse data modalities, including historical traffic data ( $X_{hist}$ ), temporal features ( $F_{temp}$ ), static network features ( $F_{static}$ ), and dynamic real-valued features ( $F_{dynamic}$ ). These are transformed into unified numerical representations via embedding layers (for continuous and categorical data) and augmented with sinusoidal positional encodings to preserve sequence order.
Customized Transformer with Low-Rank Adaptation (LoRA):
The core prediction engine is an Encoder-Decoder Transformer.
- Encoder: Processes historical traffic and multi-modal features. Crucially, the authors integrate Low-Rank Adaptation (LoRA) directly into the encoder during the initial training phase, rather than for fine-tuning a pre-trained foundation model. The weight update $\Delta W$ is approximated by the product of two low-rank matrices $A$ and $B$ ( $\Delta W \approx AB^T$ ). This acts as a regularizer against noisy data and drastically reduces the number of trainable parameters (by over 90% in the reported configuration).
- Decoder: Predicts future traffic values. It receives the encoder's output fused with external urban mobility and congestion features.
Multimodal Fusion and Causality:
A robust fusion strategy injects external mobility features into the decoder. To ensure strict temporal causality, these external features correspond strictly to historical time steps or are derived from autoregressive forecasts, preventing the leakage of ground-truth future information during inference.

Key Contributions
The paper outlines three primary contributions:

PEHT Architecture: A novel framework for network traffic prediction that explicitly integrates diverse urban mobility and congestion data, addressing the gap in existing methods that often overlook these external factors.
LoRA Integration: The application of LoRA within the Transformer Encoder to efficiently handle high-dimensional spatio-temporal network traffic data, significantly reducing parameter counts while maintaining accuracy.
Multimodal Fusion Strategy: A mechanism that injects real-time urban mobility features into the Transformer's decoding process, providing explicit contextual insights without violating temporal constraints.

Experimental Results
The PEHT model was evaluated on the Telecom Italia Big Data Challenge (Milan) dataset and five synthetic datasets generated via the CARLA simulator to test various congestion and mobility scenarios (e.g., base-station congestion, cell-edge congestion, vehicular-only, and pedestrian-only mobility).

Performance on Real Data: PEHT outperformed state-of-the-art baselines (including HGCRN, ST-DenseNet, STCNet, ST-Tran, MVSTGN, and ST2T). For SMS traffic, PEHT achieved an RMSE of 18.42 compared to the next best (ST2T) at 21.58, representing a 14.6% reduction. For Internet traffic, it achieved an RMSE of 122.10 versus 138.64, an 11.9% reduction. It also showed improvements in MAE and $R^2$ scores across SMS, Call, and Internet traffic categories.
Ablation Studies: Experiments on synthetic datasets demonstrated that the "Full Model" (combining Encoder Fusion and LoRA) provided the most consistent performance across RMSE, MAE, and $R^2$ metrics. While LoRA alone showed high $R^2$ in specific scenarios (e.g., cell-edge congestion), the full architecture proved superior in overall robustness.
Parameter Efficiency: The study quantified the parameter reduction, showing that for an input/output dimension of 4,372, a full weight matrix requires ~19.1 million parameters. Using LoRA with rank $r=8$ , this was reduced to approximately 69,952 trainable parameters, making the model feasible for large-scale urban deployments.

Significance and Claims
The paper claims that PEHT successfully bridges the gap between parameter efficiency and the integration of external urban data. By utilizing LoRA as a regularizer during initial training, the model effectively handles high-frequency noise in traffic data without overfitting, while the multimodal fusion strategy allows the decoder to leverage explicit insights from urban dynamics. The authors assert that this approach enables proactive optimization and resource management in dynamic cellular environments, offering a scalable alternative to parameter-heavy Transformer models and less flexible GNN approaches. The implementation is made available via a GitHub repository.