Resource-efficient Semantic Coding Schemes with Manifold-constrained Hyper-connections

Technical Summary: Resource-efficient Semantic Coding Schemes with Manifold-constrained Hyper-connections

Problem Statement
Semantic communication (SemCom) and task-oriented communication (TOC) aim to reduce wireless resource consumption by transmitting meaning or task-relevant information rather than raw bits. However, a critical challenge remains: achieving robust semantic representation under noisy and fading channels while maintaining transmission compactness. Existing learning-based transceivers often address reliability by increasing encoder size or channel feature dimensionality, which inflates computational complexity and channel usage. Furthermore, many learned systems rely on fixed-dimensional continuous features or rate surrogates (e.g., KL penalties) that do not correspond to actual entropy-coded lengths, making explicit rate control for bandwidth and power constraints difficult. Additionally, standard semantic encoders typically rely on single-stream residual connections, which may limit representation diversity in deep networks.

Methodology
The authors propose a Manifold-constrained Hyper-connection (mHC) coding scheme integrated with an Entropy Bottleneck (EB) for resource-efficient SemCom and TOC. The framework operates as an end-to-end learned joint source-channel coding (JSCC) system.

1. mHC Semantic Encoder:

   • Instead of a single residual path, the encoder utilizes Hyper-Connections (HC), which maintain $S$ parallel residual streams.
   • To prevent stream imbalance and training instability common in unconstrained HC, the authors impose a Doubly Stochastic (DS) constraint on the residual mixing matrix ($H^l_{res}$). This ensures that the mixing of streams is mass-preserving (rows and columns sum to one).
   • Pre-mixing and post-mixing weights are normalized via softmax operations.
   • The architecture includes a stream expansion phase (replicating input tokens to $S$ streams), $L$ mHC Transformer layers, and a stream reduction phase (summing streams back to a single vector) before the channel encoder. This ensures the downstream components operate on a single-stream feature set, preserving compatibility with standard pipelines.

2. Entropy Bottleneck (EB) for Rate Control:

   • The EB quantizes the normalized channel features and employs a learned entropy model to estimate the coding rate.
   • This provides a differentiable, explicit estimate of the bit rate, enabling end-to-end optimization of the rate-distortion (for SemCom) or rate-task (for TOC) tradeoff under bandwidth and power constraints.

3. Training Strategy:

   • The system undergoes a two-stage process: (1) SemCom pre-training to learn robust, rate-aware semantic representations, and (2) TOC adaptation where the transceiver is frozen, and only a lightweight task head is trained.

Key Contributions

1. Novel Encoder Architecture: The paper is the first to apply HC and mHC to SemCom/TOC transceivers. The DS-constrained multi-stream encoder enhances representation diversity and training stability with negligible parameter and computational overhead.
2. Explicit Rate Control: By integrating an EB-based entropy coder, the system achieves explicit rate-distortion/task optimization, allowing for precise control over transmission resources (bandwidth and power).
3. Theoretical Efficiency: The authors prove theoretically that DS-constrained stream mixing is entropy non-increasing. Specifically, the differential entropy of the transmitted features does not increase due to the mixing, implying no increase in the ideal entropy-coding length. Consequently, the multi-stream architecture introduces no additional channel uses after stream reduction.
4. Comprehensive Evaluation: Extensive experiments demonstrate the scheme's superiority over residual and unconstrained HC baselines across various channel conditions (AWGN, Rayleigh, Rician fading) and imperfect Channel State Information (CSI).

Experimental Results
The proposed scheme was evaluated on SemCom (text transmission using FineWeb10B and WikiText-103) and TOC (text classification using AG News).

• Performance: Under AWGN and Rayleigh fading, mHC consistently achieved lower perplexity (PPL) and higher classification accuracy compared to standard residual baselines and unconstrained HC. For instance, in a 48-layer model under Rayleigh fading at 10 dB SNR, mHC reduced PPL by 30% compared to HC and 36% compared to the baseline.
• Robustness: The scheme demonstrated strong zero-shot generalization to Rician fading and imperfect CSI without retraining.
• Resource Efficiency: The mHC mixing matrices added only ~0.05% to 0.2% of the total parameters. The computational overhead (FLOPs) was less than 3% relative to the baseline. Crucially, the transmission rate (number of channel symbols) remained unchanged compared to the baseline.
• Training Stability: mHC exhibited faster convergence and lower final loss compared to unconstrained HC. For the 6-layer model, mHC trained 41% faster than HC, attributed to the DS constraint stabilizing optimization.
• Comparison with Coding Strategies: Compared to Dense (DeepSC) and VIB-based coding, the EB-based approach provided essential rate control. Dense coding failed catastrophically under fading due to lack of quantization/rate control, while VIB suffered from posterior collapse (zero effective rate).

Significance and Claims
The paper claims that the mHC coding scheme offers a promising approach for robust and bandwidth-efficient learned wireless communication. Its primary significance lies in decoupling the need for richer semantic representations from the cost of increased transmission resources. By proving that DS-constrained mixing does not increase ideal coding length, the authors demonstrate that one can improve representation diversity and training stability without incurring additional channel uses or computational burdens. The integration of explicit entropy-based rate control further ensures that the system can operate effectively under strict bandwidth and power constraints, addressing a key limitation in current learning-based SemCom/TOC systems.