⚡ electrical engineering

Alternative Shapes of Modulation Schemes Detailed Exposition and Simulation Methodology

This paper presents a comprehensive study and large-scale simulation of diverse modulation constellation designs—including classical, lattice-based, and machine learning-assisted schemes—demonstrating that optimizing for symbol error rate alone is insufficient for energy efficiency and that joint optimization of reliability, robustness, and hardware constraints yields superior performance in realistic communication channels.

Original authors: Nipun Agarwal

Published 2026-01-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nipun Agarwal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a message across a noisy, windy field. You have a limited amount of energy (battery power) and a specific way of shouting (your voice). In the world of digital communication, this "shouting" is called modulation. It's the method we use to turn bits of data (0s and 1s) into signals that travel through the air.

For decades, engineers have used a standard "shouting style" called QAM (Quadrature Amplitude Modulation). Think of QAM like arranging your messages on a perfect square grid. It's easy to draw, easy to read, and works well in calm weather. However, the paper argues that this square grid has two big problems:

It wastes energy: The corners of the square are very far from the center. To shout from those corners, you have to scream very loudly, which drains your battery and distorts your voice (nonlinear distortion).
It breaks in the wind: If the wind (fading) blows hard, the square grid gets distorted, and the receiver might get confused.

This paper is a massive investigation into alternative ways to arrange these messages to be smarter, more energy-efficient, and tougher against the wind. The author, Nipun Agarwal, tests fourteen different "shapes" of these message grids to see which one is the best all-rounder.

The Main Characters (The Shapes)

The paper compares the old "Square Grid" against some new, creative shapes:

The Hexagon (Lattice-Based): Imagine packing oranges in a box. A square grid leaves gaps between the oranges. A hexagonal pattern packs them tighter. This paper finds that packing messages in a hexagon allows you to fit more data in the same space or send it with slightly less error.
The Spiral (Golden Angle Modulation): Instead of a grid, imagine arranging messages along a spiral, like the seeds in a sunflower. This shape is naturally smooth and doesn't have sharp "corners" that require screaming. This makes it very energy-efficient because you don't have to shout as loudly at the edges.
The Bell Curve (Probabilistic Shaping): Imagine you have a set of loud and quiet shouts. The old way shouts every volume equally. The new way shouts the quiet ones more often and the loud ones rarely. This mimics how nature works (like a bell curve) and is mathematically the most efficient way to use energy, though it requires a more complex "translator" at the receiver.
The AI-Designed Shapes: The paper also uses Machine Learning (AI) to invent its own shapes. The AI is told, "Make a shape that is hard to mess up, uses little energy, and fits in the wind." The AI then draws shapes that sometimes look like the hexagon or the spiral, proving that the math holds up even when a computer discovers it on its own.

The Big Discovery: Speed vs. Fuel

The most surprising finding in the paper is a trade-off between reliability (not making mistakes) and energy (battery life).

The Old Rule: "To get the best performance, you must minimize errors at all costs."
The New Rule: "Sometimes, making slightly more errors is worth it if it saves a huge amount of energy."

The paper shows that some shapes (like the Spiral or Hexagon) might make a few more mistakes than the perfect Square Grid in a calm room, but because they don't require the transmitter to "scream" (high Peak-to-Average Power Ratio), they save so much energy that they are actually better for the system as a whole.

Analogy: Imagine driving a car.

Square QAM is like driving a sports car at the speed limit. It's fast and precise, but it guzzles gas.
The New Shapes are like driving a hybrid car slightly slower. You might arrive a few seconds later (a tiny increase in errors), but you save a massive amount of fuel (energy). In a world where we need to save energy, the hybrid is the winner.

The Wind Test (Fading Channels)

The paper also tests these shapes in a "windy" environment (Rayleigh fading), which simulates real-world signal interference.

The Square Grid gets blown around easily.
The Spiral and Constant-Envelope shapes (which shout at a steady volume) are much more stable in the wind. They don't get distorted as easily, making them more robust for real-world use.

The Conclusion: One Size Does Not Fit All

The paper concludes that there is no single "perfect" shape. The best choice depends on your situation:

If you have a strong battery and a calm connection: Stick with the classic Square Grid (QAM).
If you are on a battery-powered device or in a windy area: Use the Spiral (Golden Angle) or Hexagonal shapes.
If you want the absolute most data possible: Use the Bell Curve method (Probabilistic Shaping), but be ready for complex hardware.

The Final Takeaway:
Designing how we send data isn't just about making the signal perfect; it's about balancing accuracy, energy, and hardware limits. The paper provides a "menu" of shapes, showing engineers that by changing the geometry of their signals, they can build communication systems that are greener, tougher, and smarter.

Technical Summary: Alternative Shapes of Modulation Schemes

Problem Statement
Digital modulation constellation design faces a critical challenge in modern communication systems: the optimality of classical schemes, such as square Quadrature Amplitude Modulation (QAM) and Phase-Shift Keying (PSK), degrades under realistic constraints. While these classical constellations are analytically tractable and widely deployed, they suffer from a "shaping loss" of approximately 1.53 dB relative to the theoretical capacity of the Additive White Gaussian Noise (AWGN) channel. Furthermore, their high Peak-to-Average Power Ratio (PAPR) necessitates significant power amplifier (PA) back-off, drastically reducing transmitter energy efficiency. These limitations are exacerbated in systems operating near saturation, such as millimeter-wave transmitters and massive MIMO arrays, and under fading conditions where uniform constellations may be penalized more severely than optimized asymmetric designs. Existing literature often evaluates constellation families in isolation or focuses solely on Symbol Error Rate (SER) or Mutual Information, neglecting the critical interplay between error performance, PAPR, and transmitter energy consumption.

Methodology
The paper presents a unified investigation of fourteen modulation constellations spanning classical, lattice-based, probabilistically shaped, geometrically shaped (including Golden Angle Modulation), and optimization-driven designs. The evaluation employs a large-scale, reproducible Monte Carlo simulation framework under both AWGN and Rayleigh fading channels.

Key methodological components include:

System Model: A discrete-time complex baseband model with perfect Channel State Information (CSI) at the receiver, utilizing Maximum-Likelihood (ML) detection.
Performance Metrics: Beyond standard SER analysis, the study incorporates a rigorous energy-consumption model. This model links constellation PAPR to PA efficiency, calculating the energy required per successfully delivered symbol.
Optimization Techniques: The paper utilizes Particle Swarm Optimization (PSO) and Genetic Algorithms (GA) to discover irregular constellation geometries tailored to specific objective functions (SER, PAPR, mutual information).
Machine Learning Framework: An autoencoder-based approach is explored where the transmitter and receiver are modeled as a deep neural network. The loss function is explicitly designed to be energy-aware, incorporating PAPR regularization and PA efficiency models alongside SER minimization.
Statistical Rigor: Simulations use $10^6$ symbols per SNR point with 95% confidence intervals to ensure statistical reliability across a wide SNR range (-5 to 50 dB).

Key Contributions

Unified Comparative Framework: The paper provides a systematic comparison of classical, lattice-based, probabilistic, geometric, and ML-optimized constellations under a single analytical and simulation framework, addressing the lack of unified reproducibility in prior studies.
Energy-Aware Evaluation: It introduces an explicit energy-consumption model that demonstrates that SER-optimal constellations are not necessarily energy-optimal. The study quantifies how modest degradations in error performance can yield substantial energy savings through reduced PAPR and improved PA efficiency.
Hypothesis Validation: The investigation validates four core hypotheses:
- Properly shaped non-uniform constellations can achieve lower SER than square QAM at the same average energy.
- Reduced PAPR yields tangible energy-efficiency gains at the transmitter.
- Optimization-based and ML-assisted designs can outperform analytically designed constellations under practical constraints.
- Rayleigh fading penalizes uniform constellations more severely than optimized asymmetric designs.
Machine Learning Integration: The paper demonstrates that ML-assisted constellation design provides a flexible framework for jointly optimizing reliability, robustness, and energy efficiency by embedding channel statistics and hardware models directly into the learning objective.

Results

AWGN Performance: Lattice-based constellations (specifically hexagonal) and GA-optimized designs consistently achieve the lowest SER at moderate-to-high SNRs, outperforming square QAM. Probabilistic shaping offers significant gains in mutual information but may incur rate loss in finite-length implementations.
Fading Robustness: Under Rayleigh fading, the performance ranking shifts. Constant-envelope and amplitude-compact constellations (e.g., Disc-GAM, Star-Shaped) exhibit greater robustness compared to designs optimized purely for AWGN distance.
PAPR and Energy Efficiency: A fundamental trade-off is identified between minimum Euclidean distance ( $d_{min}$ ) and PAPR. While Bell-GAM and Gaussian-shaped designs offer high shaping gains, their high PAPR leads to significant energy penalties due to PA inefficiency. Conversely, Disc-GAM and PSK-based schemes offer a favorable balance, achieving lower energy consumption per successful symbol despite potentially higher SER in AWGN.
Composite Scoring: A composite scoring metric (weighting $d_{min}$ , PAPR, and SER) reveals that lattice-based designs offer the best overall balance, though 16-QAM remains highly competitive due to implementation simplicity.
ML Findings: ML-based optimization successfully rediscovered known structures (e.g., hexagonal lattices under AWGN, disc-like geometries under energy constraints), validating classical theory while extending it to complex, multi-objective scenarios.

Significance and Claims
The paper claims to establish modulation design as a multi-objective optimization problem rather than a single-metric pursuit. Its primary significance lies in shifting the paradigm from purely error-rate-centric design to energy-aware design. The authors assert that:

SER-optimal is not energy-optimal: Designs minimizing SER under AWGN may be energy-inefficient when PA behavior is considered.
PAPR is a first-class design metric: Modulation design must explicitly account for PAPR, particularly in power-limited and green communication systems.
Context-Dependent Optimality: No single modulation scheme is universally optimal. The "best" choice depends on the operating regime (AWGN vs. fading), energy constraints, and hardware characteristics.
Practical Guidelines: The study culminates in design guidelines mapping operating regimes to appropriate strategies: low-PAPR constellations for power-limited systems, probabilistic shaping for high-throughput linear links, and ML-assisted designs for adaptive, hardware-aware optimization.

The work concludes that while classical schemes remain essential baselines, future next-generation wireless systems require intelligent, energy-aware modulation strategies that jointly optimize geometric, probabilistic, and hardware-aware constraints.

The Main Characters (The Shapes)

The Big Discovery: Speed vs. Fuel

The Wind Test (Fading Channels)

The Conclusion: One Size Does Not Fit All

More like this