💻 computer science

Unwarping the Lens: A Physics-Grounded Approach to Video Glasses Removal

This paper proposes JFSnet, a physics-grounded video glasses removal framework that transfers multi-view knowledge from a commercial generative model via synthetic data augmentation and optical simulation to achieve high-fidelity, temporally consistent restoration while preventing identity drift.

Original authors: Radim Spetlik, David Futschik, Radek Danecek, Feitong Tan, Ziqian Bai, Rohit Pandey, Yinda Zhang

Published 2026-08-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Radim Spetlik, David Futschik, Radek Danecek, Feitong Tan, Ziqian Bai, Rohit Pandey, Yinda Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of digital images, faces are the most recognizable landmarks we know. We instantly identify friends, celebrities, and strangers by the unique arrangement of their eyes, nose, and mouth. However, when a person wears eyeglasses, a complex optical barrier stands between the viewer and the true face. Lenses are not merely flat pieces of plastic; they are curved surfaces that bend light, magnifying or shrinking the features behind them, while the frames cast shadows and create sharp reflections. For decades, computer scientists have tried to teach machines to remove these glasses from photos and videos, hoping to reveal the face underneath. Early attempts often relied on guessing what the hidden face might look like, using vast libraries of human faces to fill in the blanks. While these methods could sometimes produce a plausible-looking eye, they frequently failed to preserve the person's true identity, accidentally changing their expression, shifting their gaze, or blurring the fine details that make a face unique. The challenge has been to remove the glass without losing the person.

A team of researchers has now developed a new approach that treats the removal of eyeglasses not as a guessing game, but as a physics problem. Instead of asking an artificial intelligence to imagine what lies beneath the lenses, they built a system that understands how light actually behaves when it passes through glass. The researchers created a pipeline that first generates thousands of synthetic pairs of faces: one set showing a person without glasses, and another showing the same person wearing them. To ensure these pairs are perfectly aligned, they used a sophisticated filtering process to discard any images where the person's pose or expression drifted even slightly. Then, rather than relying solely on these generated images, they simulated the actual physics of light refraction. They calculated how a specific lens would bend the light from the person's eye, creating a realistic distortion that the computer could learn to reverse. This process allowed them to train a specialized network, which they call a joint feature-spatial network, to act as a digital undo button for eyewear.

The results of this method are a significant step forward in how machines understand and edit video. When tested on thousands of high-resolution portraits, the system successfully removed glasses while keeping the person's identity intact, achieving a level of detail that previous methods could not match. In video sequences, where the person moves and the light changes, the system maintained a steady, flicker-free result, preserving the exact direction of the person's gaze and the subtle movements of their facial muscles. In direct comparisons with other advanced tools, including commercial video editors and image-based generators, this new approach was consistently preferred by human observers for its ability to keep the face looking like the original subject. The system operates with remarkable speed, processing video at nearly twenty-eight frames per second, which is fast enough for real-time applications.

The researchers also demonstrated that their method avoids the common pitfalls of earlier technologies. Many previous systems would hallucinate new features, such as changing a person's eye color or altering the shape of their smile, because they were trying to generate a new face from scratch. By grounding their training in the physical laws of optics, the new system learned to simply reverse the distortion caused by the lens, revealing the face that was already there. This distinction is crucial: the system does not invent a new person; it uncovers the one that was obscured. The study confirms that by combining the creative power of large-scale image generators with the rigid rules of physics, it is possible to achieve a level of fidelity that was previously out of reach. The work suggests that the future of digital editing lies not just in generating new content, but in understanding the physical reality of the light that creates it.

Technical Summary: Unwarping the Lens: A Physics-Grounded Approach to Video Glasses Removal

Problem Statement

High-fidelity removal of eyeglasses from video presents a significant challenge in facial attribute editing. Unlike standard occlusions, eyeglasses are complex optical elements that introduce depth-dependent refractive distortions and view-dependent specular reflections, often obscuring the underlying facial geometry. Existing approaches relying on stochastic generative priors (e.g., GANs or diffusion models) struggle to separate these optical effects from the true facial structure. Consequently, these methods frequently suffer from "identity drift," where the restored subject's expression, pose, or unique structural features diverge from the original. Furthermore, frame-by-frame application of such models often leads to temporal flickering due to a lack of constraints across the temporal domain.

Methodology

The authors propose a novel transfer framework that bridges large-scale generative priors with physical restoration principles. The pipeline consists of three main components: data curation, physics-based augmentation, and a specialized restoration architecture.

1. Generative Data Creation and Filtering

The training data is synthesized using Nano Banana (Gemini 3 Pro Image), a commercial-grade generative model.

Synthesis: For each synthetic identity, the model generates 13 high-fidelity image sets capturing diverse head poses and expressions, both with and without eyeglasses.
Alignment: To address spatial misalignment inherent in generative models, the authors employ an As-Rigid-As-Possible (ARAP) deformation mesh. This aligns the "clean" face to the exact pose and configuration of the "glasses-wearing" image using landmark constraints and multi-scale photometric loss.
Filtering: A rigorous three-stage filtering process ensures data quality:
1. L1 Filtering: Removes samples with significant shifts in head pose, gaze, or identity via background and ocular error analysis.
2. Structural Consistency: Filters out identities where the generative model introduced unwanted expression drift.
3. Reconstruction Proxy: A lightweight U-Net performs preliminary restoration; identities with high reconstruction error (indicating residual misalignments or artifacts) are discarded.
  This process yields a curated dataset of 1,860 identities (24,180 image pairs).

2. Physics-Grounded Training Augmentation

To overcome the simplicity bias of generative models regarding optical effects, the authors introduce a physics-based augmentation pipeline that operates during training:

Refraction Simulation: Using 3D parametric face fitting and monocular depth estimation, the system simulates the optical properties of lenses. Rays are cast through spherical lens surfaces defined by target diopter strengths (using Vogel's Rule), and Snell's law is applied to calculate the refractive warp. This creates a dense warp field ( $W$ ) that simulates depth-dependent magnification/minification.
Reflection Simulation: Specular reflections are generated using inverse-view environment maps conditioned on the subject's portrait, simulated in High Dynamic Range (HDR) to capture realistic highlights and variable lens coatings.
Training Input: The final training input $I$ is a composition of the generative glasses image and the physically simulated clean face:
$I = (1 - M_{lens}) \cdot G_{glasses} + M_{lens} \cdot (\mathcal{W}(G_{clean}) + \mathcal{R}) + \epsilon$
where $M_{lens}$ is the lens mask, $\mathcal{W}$ is the refractive warp, and $\mathcal{R}$ is the reflection layer.

3. JFSnet Architecture

The restoration network, JFSnet (Joint Feature-Spatial network), operates directly in pixel space to preserve high-frequency details often lost in latent diffusion approaches.

Encoder: Utilizes a pre-trained DINOv2 (ViT-L/14) encoder to capture stable semantic identity priors invariant to local distortions. Only the last four layers are fine-tuned.
Decoder: A ResNet-based convolutional decoder reconstructs fine facial features.
Skip Connections: Direct skip connections from the input image bypass the information bottleneck, ensuring the network retains details not occluded by the glasses.
Temporal Consistency: To prevent flickering without expensive flow estimation, the model enforces translation equivariance during training. A loss term ( $\mathcal{L}_{temp}$ ) forces the network to commute with small spatial shifts, ensuring restored features move coherently with the subject.

Key Contributions

Deterministic Transfer Framework: The authors formulate video glasses removal as the offline transfer of a generative prior into a deterministic, feed-forward model, combining curated generative sets with physics-based augmentation.
JFSnet Architecture: A hybrid ViT-CNN architecture that integrates semantic features with spatial reconstruction, enabling identity preservation and temporal consistency.
Physics-Based Augmentation: A novel pipeline that simulates realistic lens refraction and reflections, allowing the model to learn to invert specific optical distortions rather than relying solely on hallucination.
Comprehensive Evaluation: Extensive evaluation on curated FFHQ and CelebV-Text datasets, demonstrating superior performance in gaze preservation, temporal stability, and structural fidelity compared to state-of-the-art baselines.

Results

Quantitative Performance: On the FFHQ dataset (12,163 images), the method achieves a Fréchet Inception Distance (FID) of 0.379 and a landmark L2 error of 0.632, outperforming baselines like Nano Banana, ProPainter, and various diffusion models.
Inference Speed: The model operates at 27.68 FPS, suitable for real-time video applications.
Perceptual Studies: In user studies on CelebV-Text video sequences, the approach was consistently preferred over diffusion and GAN-based baselines (including Runway Gen-4.5) for gaze consistency, temporal stability, and overall restoration quality.
Ablation: Experiments confirm that both refraction/reflection simulations and the translation-equivariance loss are critical for achieving low FID scores and temporal stability.

Significance

The paper claims to bridge the gap between large-scale generative priors and physical restoration. By transferring the photo-realistic, multi-view knowledge of a generative model into a specialized architecture trained on physically simulated data, the approach achieves high-fidelity structural accuracy while maintaining real-time performance. The work addresses the specific limitations of stochastic priors in preserving identity and expression, offering a solution that respects the geometric and optical constraints of the human face. The authors position this as a step forward in precise local editing, moving beyond global style transfer to handle complex optical occlusions with deterministic reliability.