💻 computer science

Deployment-Aware Codec Selection for Learned RGB-D Compression in Edge--Cloud 3D Reconstruction

This paper proposes a deployment-aware framework for selecting learned RGB-D codecs in edge-cloud 3D reconstruction systems, demonstrating that prioritizing explicit hardware and runtime constraints over offline rate-distortion performance yields a practical solution that balances encoding speed, memory usage, and reconstruction quality better than traditional hardware baselines.

Original authors: Yiliu Zhang, Zili Zhang, Ziqiong Zhang

Published 2026-09-23

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiliu Zhang, Zili Zhang, Ziqiong Zhang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world of digital imaging, cameras do more than just capture a flat picture of the world. Many devices, from smartphones to specialized sensors, now record two things at once: the color of a scene and the distance to every point within it. This combination, known as RGB-D data, creates a rich, three-dimensional map that allows computers to understand the shape and layout of a room, a forest, or a street. However, this extra layer of depth information comes with a heavy price: it doubles the amount of data that needs to be stored or sent. When this data is captured on a small, battery-powered device like a drone or a robot and needs to be sent to a powerful computer in the cloud for processing, the challenge becomes how to shrink the file size without losing the critical details needed to rebuild the 3D shape later.

For years, engineers have tried to solve this by finding the perfect balance between file size and image quality. They have developed advanced computer programs, often called learned codecs, that use artificial intelligence to predict and compress images more efficiently than traditional methods. The standard approach has been to look for the algorithm that produces the smallest file for the clearest picture, assuming that if it works well in a computer simulation, it will work well in the real world. But this assumption often fails when the data must travel from a tiny, energy-constrained device to a remote server. The real world introduces physical limits: the device might not have enough memory to run a complex program, the software might not be compatible with the device's hardware, or the time it takes to compress the image might be too long for a live video feed. A method that looks perfect on a graph can become useless if it cannot actually run on the device that needs to send the data.

A team of researchers at the Harbin Institute of Technology and the Dalian University of Technology set out to fix this disconnect. Instead of just looking for the best compression numbers, they designed a new way to choose which compression tool to use. They treated the ability to actually run the software on a specific device as a primary requirement, just as important as the quality of the picture. Their goal was to build a complete system where a camera on an edge device captures a 3D scene, compresses it, sends it over a network, and has it reconstructed into a colored 3D point cloud on a cloud server, all while respecting the strict limits of the hardware.

The researchers began by testing four different types of AI-based compression models on a standard dataset of indoor scenes. They measured how small the files got and how clear the images remained. As expected, the most complex models, which used sophisticated mathematical structures to predict pixel values, produced the smallest files with the highest quality. One of these advanced models, which used a technique called attention to focus on important details, achieved a very high quality score at a very low data rate. However, when the researchers looked at how long these models took to run on a real device, the picture changed. The most powerful model was incredibly slow, taking a long time to process each frame because it had to calculate values in a strict, step-by-step sequence that the device's hardware could not speed up.

The team then applied their new selection method, which filters out any model that cannot meet specific real-world constraints. They looked for models that could run within a set time limit, fit within the device's memory, and work with the specific software tools available on the device. They found that the fastest, most efficient model was actually a simpler one. This model used a straightforward mathematical approach that the device's hardware could handle very quickly. While this simpler model produced slightly larger files and slightly lower image quality than the complex one, it was vastly faster. In fact, the complex model was more than sixty times slower to prepare for deployment than the simpler one. The researchers concluded that for a real system, the "best" model is not the one with the highest theoretical quality, but the one that can actually run fast enough to keep up with the camera.

To prove this, the team built a complete working system using a small, powerful computer board called the Jetson TX2 to act as the edge device. They programmed it to capture color and depth images, compress them using their chosen simpler model, and send the data over a local network. On the receiving end, a cloud server decoded the data and turned it back into a 3D point cloud. They measured every step of this journey, from the moment the camera took the picture to the moment the 3D shape appeared on the screen. The system successfully delivered a reconstructed 3D view at a rate of nearly thirty frames per second, with a total delay of just over ninety milliseconds. This speed is fast enough for many interactive applications, such as remote telepresence or real-time mapping.

The researchers also compared their new system against established, traditional compression tools that are already built into hardware. These older tools, which handle color and depth separately, were much faster, compressing images in just over ten milliseconds. However, they required slightly more data to achieve a similar level of quality. The new AI-based system, while slower, managed to produce a 3D reconstruction with fewer errors in the depth measurements, meaning the 3D shapes were more accurate. This trade-off showed that the new method could be a viable alternative when the priority is the accuracy of the 3D shape rather than the absolute fastest processing speed.

A key part of their success was how they managed the software on the device. They did not try to force the entire complex AI program to run on the device's specialized graphics processor. Instead, they split the work. The main parts of the program that transform the image were converted to run efficiently on the device's hardware, while the final step of packing the data into a stream was handled by a different, compatible software layer. This hybrid approach allowed them to get the speed benefits of the hardware without getting stuck on parts of the software that the hardware could not handle. They found that this specific way of organizing the software was crucial; trying to run the entire program in a single, unoptimized way would have been too slow.

The study also looked at how the data behaved when it traveled over the network. They sent the compressed images in small packets, similar to how video streaming services work, and checked how the system handled delays or lost data. They found that the system was robust, able to reassemble the images correctly even when the network conditions were not perfect. The total time it took for a picture to go from the camera to the final 3D reconstruction on the cloud was measured to be about ninety-one milliseconds on average. This includes the time to capture the image, compress it, send it, and rebuild it. The researchers noted that the biggest variation in this time came from the network transmission itself, which is expected, but the system remained stable and consistent.

Ultimately, the researchers demonstrated that choosing a compression tool is not just a math problem of finding the smallest file. It is a system design problem that must account for the physical limits of the device, the speed of the hardware, and the requirements of the final task. By treating the ability to deploy the software as a core part of the decision, they were able to select a model that balanced quality, speed, and resource usage effectively. Their work provides a clear blueprint for how to build these edge-to-cloud systems, showing that the best solution is often the one that fits the machine, not just the one that wins a theoretical contest. The result is a practical, working system that can capture, compress, and reconstruct 3D scenes in real-time, bridging the gap between advanced artificial intelligence and the physical constraints of the devices that carry them.

Technical Summary: Deployment-Aware Codec Selection for Learned RGB-D Compression in Edge–Cloud 3D Reconstruction

Problem Statement

Edge–cloud multimedia pipelines integrate sensing, compression, transmission, and downstream tasks like 3D reconstruction across heterogeneous devices. In such systems, selecting a learned codec based solely on offline rate–distortion (RD) performance or a scalarized rate–distortion–complexity (RDC) objective is insufficient. This is because deployment constraints—such as entropy-model seriality (e.g., autoregressive dependencies), runtime compatibility (e.g., TensorRT operator support), memory budgets, and the architectural decision of where to place reconstruction (edge vs. cloud)—are heterogeneous. They cannot be faithfully expressed as a single weighted penalty without obscuring the specific reasons a candidate is or is not deployable. Consequently, a codec that appears optimal in an offline PSNR–bpp plot may fail in a real-world edge–cloud pipeline due to execution latency, unsupported graph operations, or incompatible entropy structures.

Methodology

The paper proposes a two-stage, deployment-aware decision framework that treats deployability as a co-equal system objective rather than a subjective scalar penalty.

Stage 1: Numerical Pareto Screening:
- Aligned RGB and depth data are represented as a joint four-channel tensor.
- Four learned codec families (factorized prior, scale hyper-prior, joint autoregressive hierarchical prior, and an attention-enhanced model) are evaluated on the SUN RGB-D dataset.
- Candidates are screened in the rate–distortion–coding-time space using actual total RGB-D coded bits per pixel (bpp) and normalized-tensor PSNR.
- Non-dominated candidates are identified based on bitstream rate, reconstruction quality, and pre-deployment coding time.
Stage 2: Explicit Feasibility Filtering:
- Non-dominated candidates are filtered by explicit, non-scalar deployment constraints:
  - Bounded encoding latency: Must meet a target frame rate (e.g., 30 FPS, $\le$ 33.3 ms).
  - Memory budget: Must fit within GPU and system RAM limits.
  - Static runtime compatibility: Must support a direct TensorRT static-graph path without custom operator plugins.
  - Entropy-model structure: Must avoid autoregressive seriality that hinders parallel execution.
  - Edge–cloud placement: The edge device must not be burdened with dense point-cloud construction; this is reserved for the cloud.
- The final selection uses a transparent tie-breaking rule, prioritizing the highest quality among feasible candidates.
System Implementation:
- Architecture: The edge device (NVIDIA Jetson TX2) captures RGB-D, aligns them, compresses the four-channel tensor, and packets the bitstream via RTP. The cloud backend decodes the stream and projects the data into a colored point cloud.
- Runtime Optimization: The selected codec (a ReLU-based factorized-prior variant) is implemented using a hybrid TensorRT–PyTorch path. Neural transforms are accelerated via TensorRT (FP16/FP32), while entropy coding remains in PyTorch.
- Baselines: The system is compared against hardware-accelerated H.264 and HEVC all-intra baselines (encoding RGB and depth separately) under matched frame-independent definitions.

Key Contributions

Feasibility-First Pareto Framework: A selection procedure that separates numerical RD trade-offs from categorical architectural constraints, avoiding the pitfalls of scalarized RDC optimization.
Cross-Layer Architecture: A design where regular-grid RGB-D data is coded at the edge, and calibrated 3D geometry is generated post-decoding in the cloud, avoiding resource-intensive point-cloud construction on the edge.
Empirical Separation of Frontiers: Demonstrates a measurable gap between the "offline compression frontier" (best PSNR) and the "deployment frontier" (best feasible performance), explaining this via probability model structures (factorized vs. autoregressive/attention).
Hybrid Runtime Benchmarking: Evaluates the selected codec on Jetson TX2 across native PyTorch, ONNX Runtime, and TensorRT, explicitly including entropy-coded stream generation in the latency measurement.
Comprehensive System Evaluation: Provides end-to-end metrics including actual bitstream bpp, edge resource/energy consumption, RTP packetization overhead, and capture-to-reconstruction latency under various network conditions.

Results

Codec Selection: While the attention-enhanced model achieved the strongest offline point (38.2 dB at 0.92 bpp), it was rejected due to high serial coding costs (approx. 64.3 $\times$ slower than the selected model). The ReLU-based factorized-prior variant was selected, achieving 36.42 dB at 1.17 bpp.
Performance vs. Baselines:
- At the deployed rate (~1.17–1.20 bpp), the selected learned codec outperformed hardware baselines in reconstruction quality (36.42 dB vs. 35.0 dB for H.264 and 35.9 dB for HEVC) and depth accuracy (lower RMSE and point-to-point error).
- However, hardware baselines were significantly faster (10.7 ms and 13.5 ms latency) compared to the learned codec (28.9 ms).
Runtime Efficiency: On the Jetson TX2, the TensorRT FP16 configuration achieved 34.60 FPS for the bounded encoding path, a 12.7 $\times$ improvement over native PyTorch CPU. It also reduced energy consumption to 0.26 J/frame (vs. 2.72 J/frame for CPU) and lowered memory usage.
End-to-End Latency: The complete LAN pipeline achieved a mean capture-to-reconstruction latency of 91.3 ms (95th percentile: 109.4 ms), with the encoding path contributing ~29.2 ms.

Significance and Claims

The paper claims that the preferred codec for edge–cloud systems changes fundamentally when deployability is treated as a primary constraint rather than a secondary complexity penalty. The proposed framework links codec structure, embedded execution, byte-level rate reporting, and cloud-side reconstruction in a reproducible, cross-layer selection procedure.

The authors emphasize that their contribution is not merely a new compression algorithm, but a selection methodology that reveals how architectural constraints (like autoregressive dependencies) can render high-performance offline models unsuitable for embedded deployment. The study validates that a slightly less efficient but structurally simpler model (ReLU-based factorized prior) is the optimal choice for this specific edge–cloud pipeline, offering a better trade-off between reconstruction quality and the ability to execute within strict hardware and latency budgets. The paper concludes that this two-stage logic is generalizable to other edge multimedia tasks (e.g., segmentation, AR rendering) by swapping the quality metric and constraints while retaining the decision structure.