Task-Adaptive Decoder Specialization for Unified Image Coding Toward Human Reconstruction and Machine Classification

Technical Summary: Task-Adaptive Decoder Specialization for Unified Image Coding

Problem Statement
The paper addresses a critical challenge in unified image coding: the difficulty of transmitting a single bitstream that simultaneously satisfies the conflicting requirements of human perception (requiring high perceptual fidelity and texture detail) and machine vision (requiring strong semantic discriminability for tasks like classification). Existing unified coding methods, which rely on shared latent representations and generic decoding paths, often struggle at low bitrates. In these scenarios, the shared representation tends to produce over-smoothed reconstructions that attenuate fine-grained textures (hurting human vision) and weaken task-relevant semantic cues (hurting machine classification). While previous approaches have attempted to share encoders or use multi-path decoding, they often fail to resolve the multi-objective optimization conflicts inherent in a single shared latent space, particularly when semantic cues are degraded by quantization.

Methodology
The authors propose a Task-Adaptive Decoder Specialization Framework built upon a shared Transformer-based learned image codec. The architecture maintains a unified encoder, entropy model, latent bitstream, and principal decoder backbone to ensure a single bitstream is transmitted. However, it introduces conditionally activated decoder-side adapters that dynamically reconfigure the decoding path based on the target task (Human Reconstruction or Machine Classification) without requiring additional transmitted bits.

The framework employs a two-stage training strategy:

1. Stage 1: The shared codec backbone is pre-trained to learn a compact, unified latent representation.
2. Stage 2: The shared backbone is frozen. Only the task-specific decoder-side components are fine-tuned under specific task conditions.

The framework integrates two specialized modules:

1. Latent Space Semantic Preservation (LSSP) for Machine Classification:

   • Location: Decoder input.
   • Function: Reshapes quantized latent features to recover semantic discriminability lost during low-bitrate compression.
   • Components:
     • Task-Quality-Aware Semantic Attention Module (TQ-SAM): A three-stage cascade (Global Channel Modulation, Multi-scale Spatial Attention, and Tiny Window MHSA) that adaptively recalibrates latent features based on quality levels and task objectives. It uses quality embeddings to modulate channel weights and spatial attention, strengthening damaged channels at low bitrates.
     • Multitask Semantic Head (MSH): A lightweight MLP attached to the output of TQ-SAM that provides direct semantic supervision (via classification loss) in the latent space. This creates a short gradient path, bypassing the frozen decoder to ensure the latent representation aligns with classification decision boundaries before pixel reconstruction.

2. Hierarchical High-Frequency Restoration (HFR) for Human Reconstruction:

   • Location: Decoder output.
   • Function: Compensates for high-frequency residuals and texture smoothing caused by the shared decoding path.
   • Mechanism: An edge-guided residual compensation design. It takes the base reconstruction and its Laplacian edge map as inputs.
   • Architecture: A three-branch multi-scale architecture captures local textures and global structures, utilizing Swin Transformer blocks for non-local contextual modeling.
   • Gating: A content-aware spatial gating mechanism generates a mask to selectively apply restoration only to texture-salient regions, avoiding artifacts in smooth backgrounds.
   • Supervision: Trained with an explicit high-frequency residual loss ($L_{res}$) using a fixed Laplacian filter, ensuring the module focuses on recovering suppressed structural details rather than uniformly sharpening the image.

Key Contributions

1. Unified Framework with Specialized Decoding: The paper proposes a framework that preserves a shared encoder and bitstream while enabling on-demand specialization of the decoding graph via conditionally activated adapters, effectively mitigating conflicts between human and machine vision objectives.
2. LSSP Mechanism: Introduction of a latent-space specialization mechanism that combines quality-aware attention and direct latent-space semantic supervision to enhance classification performance under compact bitstreams.
3. HFR Module: Design of a hierarchical, non-local restoration network that recovers high-frequency structural textures without additional bitrate costs, addressing the over-smoothing issue in generic reconstruction paths.
4. Two-Stage Optimization: A training strategy that freezes the shared backbone to prevent task interference during fine-tuning, allowing for efficient adaptation to specific tasks.

Experimental Results
The method was evaluated on standard datasets (ImageNet-1K for classification; Kodak and DIV2K for reconstruction) and compared against State-of-the-Art (SOTA) unified coding methods (MPA, MRIC, CRDR) and task-specific approaches (TransTIC, SFT).

• Machine Classification: On ImageNet-1K, the proposed method achieved 77.86% Top-1 accuracy at 0.15 bpp, outperforming MPA by 1.69 percentage points and TransTIC by 8.61 percentage points. It demonstrated an 82.17% BD-Rate saving and a 9.20% BD-Acc gain relative to the VTM17.1 anchor.
• Human Reconstruction: On the Kodak dataset, the method achieved 29.78 dB PSNR at 0.15 bpp, surpassing MPA, MRIC, CRDR, and TinyLIC. Relative to the HiFiC anchor, it achieved a 54.23% BD-Rate reduction and a 2.81 dB BD-PSNR gain.
• Ablation Studies:
  • Removing LSSP components resulted in significant drops in classification accuracy and BD-Rate savings, validating the necessity of both TQ-SAM and the direct MSH supervision path.
  • Replacing the HFR Swin-Transformer with a simple CNN yielded negligible improvements, confirming the importance of non-local modeling. Adding explicit high-frequency residual supervision ($L_{res}$) provided an additional ~1.1 dB gain at low bitrates, proving the value of frequency-domain guidance over standard MSE optimization.

Significance and Claims
The paper claims that its primary significance lies in resolving the trade-off between perceptual fidelity and semantic discriminability within a single unified bitstream without retraining the shared encoder or increasing transmission overhead. By shifting the specialization burden to the decoder side via lightweight, conditionally activated adapters, the framework avoids the parameter redundancy of separate coding schemes and the optimization conflicts of purely shared latent representations. The authors assert that this approach is particularly effective in the low-bitrate regime, where generic decoding paths typically fail to preserve the fine-grained details and semantic cues required for both human viewing and machine analysis. The work suggests that task-adaptive decoder specialization is a viable path toward efficient, unified human-machine vision systems.