Why Channel-Centric Models are not Enough to Predict End-to-End Performance in Private 5G: A Measurement Campaign and Case Study

This paper demonstrates that channel-centric models, including ray-tracing simulators, fail to accurately predict end-to-end throughput in private 5G networks due to systematic over-estimation of MIMO spatial layers, whereas data-driven Gaussian process models trained on direct measurements provide significantly more reliable predictions for communication-aware robot planning.

The Gist

Here is an explanation of the paper, translated into everyday language with some creative analogies.

The Big Picture: The Robot's "Crystal Ball" Problem

Imagine you are programming a robot to zip around a factory, carrying heavy parts from one machine to another. To do this safely and quickly, the robot needs to talk to a central computer via a 5G network. If the connection drops, the robot might crash or stop working.

So, before the robot even moves, engineers try to predict: "If the robot goes to this spot, how fast will the internet be?"

For a long time, engineers have used two main ways to make this prediction:

1. The Physics Simulator (The "Map Maker"): A computer program that draws a 3D map of the factory and uses the laws of physics to calculate how radio waves bounce off walls and metal.
2. The Data Learner (The "Experienced Local"): A computer program that looks at real measurements taken by a robot walking around the factory and learns patterns from that data.

The Big Question: Do these methods actually tell us the truth about how fast the internet will be?

The Experiment: A Robot in a Nuclear Bunker

The researchers took this question to a very specific place: the KTH Reactor Hall in Stockholm. It's an underground, radio-shielded room that used to house a nuclear reactor. Because it's shielded and empty, it's the "perfect classroom" for testing—no outside interference, no random people walking around.

They set up a private 5G network there and sent a robot on a grid pattern to measure the actual internet speed at hundreds of different spots. Then, they compared the robot's real data against the predictions from the two methods mentioned above.

The Surprise: The "Map Maker" Got the Signal Right, But the Speed Wrong

Here is where the story gets interesting.

The Physics Simulator (The Map Maker) was actually pretty good at predicting the signal strength. It knew exactly where the walls were and where the signal would be strong or weak. It was like a weather forecaster who correctly predicted it would be sunny.

However, when it came to predicting the actual internet speed (throughput), the simulator was wildly optimistic. It kept telling the engineers, "You'll get super-fast speeds here!" But when the robot actually went there, the speed was much slower.

Why? The "Four-Lane Highway" Illusion.
Think of 5G MIMO (Multiple-Input Multiple-Output) technology like a highway with multiple lanes.

• The Simulator's View: It assumed the highway always had 4 lanes open for traffic, no matter what. It thought, "The signal is strong, so we can use all 4 lanes!"
• The Reality: In the real world, the highway was often only 1 or 2 lanes open. The system was smart enough to realize that even though the signal was strong, the "traffic" (interference and physics) was too messy to safely use all 4 lanes. So, it dropped back to 1 or 2 lanes to stay stable.

Because the simulator didn't know about these "lane closures," it kept predicting 4x the speed that was actually possible. It was like a GPS telling you, "You'll get there in 10 minutes!" because it assumes you have a 4-lane highway, while in reality, you're stuck in a single-lane construction zone.

The Solution: The "Experienced Local" (Data-Driven Model)

The researchers then tried the second method: Gaussian Process Regression (GPR).

Instead of trying to calculate physics, this method was like hiring a local guide who has walked the factory floor a thousand times.

• The robot walked around, measured the speed, and fed that data to the model.
• The model didn't care about walls or antennas; it just learned: "When the robot is here, the speed is this."

The Result:
The "Local Guide" was incredibly accurate. It predicted the speed almost perfectly, with almost no bias. It learned the messy reality of the factory floor (the "lane closures") directly from the data, bypassing the need to guess how the physics worked.

The Takeaway: Don't Trust the Map, Trust the Experience

The paper concludes with a warning for anyone building robots that rely on 5G:

1. Signal Strength $\neq$ Speed: Just because the signal is strong (good bars on your phone) doesn't mean the internet is fast. The system might be throttling itself down to stay stable.
2. Physics Simulators Lie: Even the most advanced physics simulators are too optimistic about speed because they don't understand the complex "traffic rules" of the 5G network (like how many lanes are actually open).
3. Data is King: If you want to know how fast a robot will actually move, you need real-world data, not just a pretty 3D map.

In short: If you are planning a robot's path, don't just look at the signal map. You need to know the real speed, or your robot might plan a route that looks great on paper but fails in reality.

Technical Summary

Here is a detailed technical summary of the paper "Why Channel-Centric Models are not Enough to Predict End-to-End Performance in Private 5G: A Measurement Campaign and Case Study."

1. Problem Statement

In industrial automation, mobile robots increasingly rely on communication-aware motion planning, where robot trajectories are optimized based on predicted wireless network performance. Current planning paradigms typically rely on channel-centric metrics (e.g., Received Signal Strength Indicator (RSSI), Signal-to-Interference-plus-Noise Ratio (SINR), or Path Loss) as proxies for end-to-end throughput.

The core problem addressed in this paper is the unverified assumption that favorable channel conditions (high SINR) reliably translate into high end-to-end throughput. The authors argue that in complex 5G environments, link-layer mechanisms (such as MIMO spatial layer adaptation, scheduler behavior, and protocol overhead) introduce significant discrepancies between channel quality and actual data rates. Existing models often fail to capture these dynamics, leading to overly optimistic trajectory plans that may violate reliability requirements.

2. Methodology

The study employs a single-site case study in a controlled, radio-shielded underground facility (KTH Reactor Hall) equipped with an Ericsson Private 5G (EP5G) network. The methodology compares three distinct approaches:

• Physics-Based Simulation (Ray Tracing):

  • Tool: Altair Feko using the Shooting and Bouncing Rays (SBR) method.
  • Setup: Utilized a high-fidelity 3D model of the environment and 3GPP-compliant indoor factory channel models.
  • Configuration: Configured for a 4x4 MIMO downlink, 80 MHz bandwidth, and specific TDD frame patterns.
  • Goal: To generate deterministic predictions of channel metrics (SINR, MCS) and infer throughput.

• Robotic Measurement Campaign:

  • Platform: A modified TurtleBot3 equipped with a Quectel RM500Q-GL 5G modem.
  • Protocol: The robot moved in discrete 0.5m steps, collecting ~9,000 data points (7,000 valid after cleaning).
  • Metrics: Collected physical/link-layer metrics (SINR, RSRP, MCS, Rank Indicator) via AT commands and end-to-end throughput via iperf3.
  • Environment: Controlled to eliminate external interference and dynamic obstacles, representing a "best-case" scenario for prediction accuracy.

• Data-Driven Modeling (Gaussian Process Regression - GPR):

  • Approach: Trained GPR models directly on the measured throughput data, bypassing intermediate channel abstractions.
  • Kernels: Evaluated three stationary covariance kernels: Radial Basis Function (RBF), Once-differentiable Matérn, and Rational Quadratic (RQ).
  • Noise Modeling: Employed heteroscedastic noise modeling to account for spatially varying measurement uncertainty.

• Comparative Analysis:

  • A unified framework compared predictions against ground truth using metrics for Bias (systematic over/under-prediction), Accuracy (MAE, RMSE), and Variability (Standard Deviation, Percentiles).

3. Key Contributions

1. Public Dataset: Creation and release of a spatially dense dataset containing downlink throughput, latency, and link-layer metrics from a private 5G industrial deployment.
2. Systematic Error Identification: Demonstration that physics-based simulators systematically over-predict throughput (92.4% of locations) even when channel metrics (SINR/MCS) are predicted accurately.
3. Root Cause Analysis: Identification of MIMO Spatial Layer Over-estimation as the dominant error source. The simulator assumed near-uniform 4-layer transmission, whereas real-world measurements showed substantial adaptation (often dropping to 1 or 2 layers) due to inter-stream interference and hardware limitations.
4. Data-Driven Validation: Proof that GPR models trained on end-to-end measurements achieve significantly higher accuracy than physics-based simulations without requiring explicit modeling of complex link-layer logic.
5. Kernel Comparison: Analysis showing that while mean prediction accuracy is similar across GPR kernels, the Rational Quadratic (RQ) kernel provides superior uncertainty estimation, better tracking the spatial variability of the environment.

4. Key Results

• Physics-Based Simulation Performance:

  • Bias: Severe systematic over-prediction with a median error of +144.8 Mbit/s.
  • Accuracy: Mean Absolute Error (MAE) of 145.1 Mbit/s and RMSE of 161.1 Mbit/s.
  • Cause: The simulator correctly predicted Modulation and Coding Scheme (MCS) but failed to model the reduction in spatial layers (Rank Indicator). It assumed 4 layers everywhere, while measurements showed 1–3 layers depending on location.

• Data-Driven (GPR) Performance:

  • Bias: Near-zero bias (Median error ~1.5–2.5 Mbit/s) with a balanced 50/50 split between over- and under-prediction.
  • Accuracy: Achieved a ~66% reduction in error compared to the simulator.
    • MAE reduced to ~36 Mbit/s.
    • RMSE reduced to ~49 Mbit/s.
  • Uncertainty: The RQ kernel produced uncertainty estimates that closely matched empirical measurement variability, making it suitable for risk-aware planning.

• MIMO Discrepancy:

  • The study revealed that even in Line-of-Sight (LOS) regions, the sustainable spatial rank is often lower than the theoretical channel rank due to hybrid beamforming constraints and inter-stream interference, a phenomenon not captured by standard ray-tracing tools.

5. Significance and Implications

• Paradigm Shift for Robotics: The findings challenge the prevailing assumption in communication-aware robotics that optimizing for SINR or RSSI guarantees high throughput. Planners relying solely on channel-centric models risk generating trajectories that fail to meet latency or reliability requirements.
• Limitations of Simulation: Even in a "best-case" scenario (controlled environment, detailed 3D models, no external interference), physics-based simulators fail to predict end-to-end throughput accurately due to the complexity of proprietary link-layer adaptations (scheduler logic, closed-loop feedback, beamforming trade-offs).
• Role of Data-Driven Models: GPR models offer a robust alternative for throughput prediction by learning system behavior directly from data. However, they face challenges regarding computational scalability and adaptability to environmental changes.
• Future Directions: The paper suggests a move toward hybrid approaches (combining physics priors with data-driven refinement) and emphasizes the need for end-to-end performance prediction rather than intermediate channel proxies for reliable autonomous system planning.

In conclusion, the paper establishes that favorable channel conditions do not guarantee high throughput in 5G systems due to link-layer dynamics. Accurate prediction for industrial robotics requires either extensive calibration of link-layer models or, more effectively, data-driven approaches that capture real-world system behavior.