💻 computer science

Seeing Through Smoke: Surgical Desmoking for Improved Visual Perception

This paper introduces a transformer-based surgical desmoking model trained on a large synthetic dataset and a newly curated real-world benchmark to effectively remove surgical smoke from endoscopic images, thereby enhancing visual perception and downstream tasks like depth estimation and instrument segmentation.

Original authors: Jingpei Lu, Fengyi Jiang, Xiaorui Zhang, Lingbo Jin, Omid Mohareri

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingpei Lu, Fengyi Jiang, Xiaorui Zhang, Lingbo Jin, Omid Mohareri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to perform a delicate surgery, but instead of a clear view, you are looking through a thick, swirling fog that was created by the very tools you are using. This is the reality of robotic surgery. When surgeons use electric tools to cut tissue or seal blood vessels, they create "surgical smoke" (a mix of vapor and tiny particles). This smoke clouds the camera lens, making it hard to see tiny details, which can slow down the operation or even lead to mistakes.

This paper introduces a clever solution: a digital "smoke vacuum" powered by Artificial Intelligence.

Here is the breakdown of their work, explained simply:

1. The Problem: The "Foggy Window"

Think of the surgeon's view as looking through a car windshield during a heavy fog storm.

The Old Way: Surgeons have to stop, pull out the camera, and wipe it clean, or use physical suction tubes to try to suck the smoke away. This interrupts the surgery and isn't always perfect.
The Goal: They wanted a software solution that could instantly "wipe the fog" off the digital image, just like a magic eraser, so the surgeon never has to stop.

2. The Solution: A "Smart De-Fogger"

The team built a special AI model (a type of computer brain called a Transformer) that acts like a super-smart photo editor.

How it works: Instead of just guessing what's behind the smoke, the AI uses a "physics-inspired" approach. It understands that smoke works like a veil: it blocks some light and adds a hazy glow. The AI tries to mathematically reverse this process to reveal the original, clear image underneath.
The Double Job: Not only does it predict what the clear image looks like, but it also draws a "map" of where the smoke is thickest. This is like the AI saying, "I see the fog here, and I'm removing it from there."

3. The Training Challenge: "Learning to See in the Dark"

To teach an AI to remove smoke, you usually need thousands of pairs of photos: one with smoke and one perfectly clear version of the exact same scene.

The Problem: In real surgery, you can't take a photo, then magically make the smoke disappear to take a second photo. It's impossible to get enough real-life "before and after" pairs.
The Creative Fix: The researchers built a virtual smoke factory. They took thousands of real, clear surgical photos and digitally "painted" realistic smoke over them using computer graphics. This created 80,000 fake training pairs for the AI to learn on.
The Real Data: To prove it works, they also collected the world's largest real-world dataset of 5,817 actual smoke/clear pairs from a da Vinci surgical robot.

4. The Results: Clearer Vision, Better Tools

They tested their "Smart De-Fogger" against other existing methods.

Image Quality: Their AI produced the clearest, most natural-looking images, restoring details that other methods missed or distorted.
The Twist (What it helps and what it doesn't):
- Helps: It makes it much easier for other computer programs to identify surgical tools (like spotting a scalpel in the fog). It's like turning on a bright light in a dark room; the tools stand out clearly.
- Doesn't Help (Yet): Interestingly, it didn't immediately help the computer calculate 3D depth (how far away things are). Why? Because the AI changed the texture of the image slightly, which confused the depth-sensing algorithms. It's like cleaning a foggy window so well that the reflection changes, confusing a distance sensor.

The Big Picture

This paper is a major step forward for robotic surgery. By creating a digital tool that clears the smoke instantly, they are giving surgeons a clearer view without stopping the procedure. While there are still some kinks to work out for 3D depth sensing, this technology promises to make surgeries safer, faster, and more precise by ensuring the surgeon always sees exactly what they need to see.

In short: They taught a computer to be a magic eraser for surgical smoke, turning a blurry, dangerous view into a crystal-clear window for the surgeon.

1. Problem Statement

Minimally invasive and robot-assisted surgeries (e.g., using the da Vinci system) rely heavily on endoscopic imaging. However, the use of high-energy instruments like electrocautery and vessel sealers generates surgical smoke (a mixture of water vapor, protein, and lipid byproducts). This smoke severely degrades visual perception by obscuring tissue details, prolonging operative time, and increasing the risk of surgical errors. Furthermore, smoke artifacts hinder downstream computer vision tasks essential for surgical automation, such as instrument segmentation, tool tracking, and 3D reconstruction.

Existing physical smoke evacuation methods (e.g., suction) have limitations regarding pressure control and gas composition. Digital smoke removal (desmoking) offers a real-time alternative but faces significant challenges:

Data Scarcity: There is a critical lack of large-scale, paired "smoky-to-smoke-free" training data from real surgical procedures.
Domain Gap: Synthetic data often fails to generalize to real surgical environments due to differences in smoke density, motion, and scattering properties.
Complexity: Surgical smoke induces non-uniform scattering that can mimic anatomical textures, making it difficult to distinguish from actual tissue.

2. Methodology

The authors propose an end-to-end Transformer-based surgical desmoking framework combined with a novel data generation pipeline.

A. Model Architecture

The model consists of three main components:

Backbone: A Vision Transformer (ViT-Base) initialized with weights pretrained on a large corpus of surgical images (via Masked Autoencoders). It extracts multi-scale features from intermediate blocks.
Decoder: A DPT (Dense Prediction Transformer) decoder that fuses latent representations to generate high-resolution outputs.
Physics-Inspired Desmoking Head: Instead of a standard regression head, the authors design a head based on the Atmospheric Scattering Model:
- The model assumes the input smoky image $I$ follows the equation: $I = Jt + A(1-t)$, where $J$ is the smoke-free image, $A$ is global atmospheric light, and $t$ is the transmission map.
- The network is trained to jointly predict the smoke-free image $J$ and a smoke map $S$ .
- To enforce physical consistency, the head directly regresses parameters $K$ and $B$ (derived from $t$ and $A$ ) to reconstruct $J$ via $J = KI - B + I$. The smoke map $S$ is predicted as a function of the airlight term $A(1-t)$ .

B. Synthetic Data Generation Pipeline

To address the lack of paired training data, the authors developed a pipeline to generate ~80,000 paired samples:

Source: 10,000 real smoke-free endoscopic images (in vivo and ex vivo).
Process: Artificial smoke patterns with varying densities and spatial distributions are blended with real images using alpha blending, RGB channel correction, and illumination enhancement.
Domain Randomization: To bridge the sim-to-real gap, the pipeline randomizes smoke color, transparency, and blending coefficients during generation.

C. Benchmark Dataset Creation

The authors curated the largest paired surgical smoke dataset to date:

Source: Captured using an Intuitive Surgical da Vinci system with porcine tissue specimens.
Strategy: A bidirectional capture strategy (progressive occlusion followed by progressive clearance) to ensure identical static scenes across a spectrum of smoke densities.
Scale: 5,817 high-resolution stereo image pairs (1024×1280) with associated depth maps and instrument segmentation labels.

3. Key Contributions

Novel Architecture: A transformer-based model with a physics-inspired head that jointly predicts smoke-free images and smoke maps, leveraging the atmospheric scattering model for better structural constraints.
Data Solutions:
- A scalable synthetic data generation pipeline producing 80k+ training pairs.
- The release of a new, large-scale, high-resolution benchmark dataset (5,817 pairs) with stereo depth and segmentation annotations.
Comprehensive Evaluation: Extensive testing on both public benchmarks and the new private dataset, including an analysis of the impact on downstream tasks (depth estimation and segmentation).

4. Results

Image Reconstruction Performance

Public Benchmark (De-Smoking Dataset): The proposed method achieved State-of-the-Art (SOTA) performance, outperforming existing dehazing and desmoking baselines (including CNNs and other Transformers) on both Cholecystectomy and Prostatectomy subsets.
- Cholecystectomy: SSIM 0.86, PSNR 25.51.
- Prostatectomy: SSIM 0.75, PSNR 21.71.
Private Benchmark: On the authors' high-resolution dataset, the model achieved an average SSIM of 0.73 and PSNR of 20.61. When fine-tuned on real smoke data, performance improved further (SSIM 0.75, PSNR 23.57).
Qualitative: The method successfully recovers scene content behind dense smoke with minimal color distortion and fewer artifacts compared to competitors.

Impact on Downstream Tasks

Instrument Segmentation: Desmoking improved segmentation performance (IoU increased from 81.15% to 81.81% with fine-tuning). The removal of smoke enhanced local contrast and clarified instrument boundaries.
Stereo Depth Estimation: Surprisingly, desmoking did not consistently improve depth estimation. In some cases, the "no-desmoking" baseline performed best.
- Reasoning: Foundation models (like FoundationStereo) are robust to smoke due to large-scale training. Furthermore, the desmoking process alters image statistics, potentially disrupting the left-right feature consistency required for stereo matching.

5. Significance and Conclusion

This work represents a significant step toward computer-assisted robotic surgery by addressing a critical bottleneck: visibility degradation due to smoke.

Clinical Relevance: The method enhances real-time visual perception without interrupting the surgical procedure, potentially reducing operative time and error rates.
Algorithmic Insight: The study highlights a nuanced trade-off: while desmoking benefits tasks relying on local contrast (segmentation), it may introduce artifacts that hinder geometry-based tasks (stereo depth) if not carefully integrated.
Future Outlook: The authors position digital desmoking as a vital component of surgical vision pipelines, with future work focusing on real-time deployment and tighter integration with computer-assisted interventions.

The paper effectively bridges the gap between synthetic training and real-world application, providing both a high-performing model and the necessary data infrastructure to advance the field of surgical computer vision.