💻 computer science

AnyBokeh: Physics-Guided Any-to-Any Bokeh Editing with Optical Fingerprint Transfer

AnyBokeh is a physics-guided framework that enables controllable any-to-any bokeh editing from a single image by estimating source optical characteristics via signed circle-of-confusion and disparity maps to transfer optical fingerprints and synthesize relative blur, thereby avoiding the artifacts and limitations of traditional all-in-focus reconstruction pipelines.

Original authors: Xinyu Hou, Xiaoming Li, Zongsheng Yue, Chen Change Loy

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinyu Hou, Xiaoming Li, Zongsheng Yue, Chen Change Loy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you take a photo with your phone or camera. Sometimes, you want the background to be blurry (like a professional portrait), and sometimes you want everything sharp. Usually, you have to get this right when you take the picture. But what if you took the photo and then decided, "Actually, I want that background sharper," or "I want that background even blurrier"?

That's the problem this paper, AnyBokeh, tries to solve. It's a new tool that lets you change the "blur" (bokeh) of a photo after you've taken it, no matter how the original photo was taken.

Here is how it works, broken down into simple concepts:

1. The Old Way: The "Reset Button" Problem

Previous methods tried to fix blurry photos by hitting a "Reset Button." They would try to magically turn the blurry photo into a perfectly sharp, "all-in-focus" picture first. Once they had that sharp picture, they would pretend to take a new photo with a different lens setting to create the new blur.

The Flaw: This is like trying to fix a smudged fingerprint by first erasing the whole finger and then drawing a new one. You lose the original clues. If the original photo was very blurry, the "Reset Button" often guesses wrong, creating fake details or weird artifacts. Also, if the original photo was already sharp, but you wanted to make it blurry, these old tools sometimes struggle to figure out how blurry to make it without a manual calibration step.

2. The AnyBokeh Way: The "Optical Fingerprint"

AnyBokeh takes a different approach. Instead of trying to erase the blur, it treats the blur as a clue.

Imagine every camera leaves a unique "fingerprint" on a photo based on how the lens was set (how wide the aperture was, how far away the focus was, etc.).

Step 1: The Detective Work. AnyBokeh looks at your photo and asks, "What kind of fingerprint is already here?" It estimates two things:
- The Blur Map: How blurry is each part of the photo right now?
- The Depth Map: How far away are the objects?
- By combining these, it calculates the camera's "Optical Fingerprint." This is a mathematical rule that tells the computer exactly how the original camera turned distance into blur.
Step 2: The Translation. Now, you tell the computer, "I want the background to look like it was taken with a different lens setting."
- Instead of guessing, AnyBokeh takes that original Optical Fingerprint and mathematically translates it to your new setting.
- It's like having a secret code that says, "If the original lens made a tree 5 meters away look this blurry, then a lens with a wider opening would make it look that blurry."
Step 3: The Artist. Finally, a smart AI editor looks at the original photo, the original blur map, and the new target blur map. It acts like a painter who knows exactly which parts to sharpen, which parts to keep as they are, and which parts to add new blur to. It doesn't just "blur" the whole image; it knows exactly where the focus should be.

3. The "Training Gym" (UnrealBokeh)

To teach this AI how to do such a complex job, the researchers built a massive, perfect training gym called UnrealBokeh.

Real-world photos are messy; you rarely know the exact settings of the camera or the exact distance of every object.
So, they used a video game engine (Unreal Engine) to create thousands of perfect 3D scenes. They took "photos" in this virtual world where they knew everything: the exact lens settings, the exact distance to every leaf and rock, and the exact blur.
This allowed the AI to learn the physics of light and blur perfectly before it was tested on real-world photos.

4. Why This Matters

The paper claims that AnyBokeh is better than previous methods because:

It works on anything: You can take a photo that is already blurry and make it sharper, or take a sharp photo and make it blurry, or change the focus point entirely.
No "Reset" needed: It doesn't force the image to become perfectly sharp first, which prevents the "hallucinated" fake details that other tools create.
No manual tuning: You don't have to fiddle with settings to get the blur strength right; the "Optical Fingerprint" handles the math automatically.

In short, AnyBokeh is like a magic lens that lets you rewrite the physics of a photo after you've taken it, using the clues already hidden in the image to make the changes look natural and real.

Technical Summary: AnyBokeh

Problem Statement

Depth-of-field (DoF) control is a fundamental aesthetic tool in photography, yet post-capture bokeh editing from a single image remains challenging. The core difficulty lies in two unknowns:

Source Blur State: The input image may already contain spatially varying defocus blur (some regions sharp, others blurred). A practical editor must understand this state to decide which regions to sharpen, preserve, or further defocus.
Optical Parameters: Camera parameters (focus distance, aperture, focal length, sensor size) are often incomplete or unavailable for "in-the-wild" images.

Existing methods typically suffer from an "AIF (All-In-Focus) bottleneck." They either assume the input is already sharp or force a reconstruction of an AIF image before rendering new bokeh. This approach discards useful blur cues from the source image and propagates reconstruction artifacts into the final edit. Furthermore, without complete metadata, these pipelines often require per-image, ground-truth-guided "bokeh-level calibration" to match the desired blur strength, which is impractical for blind editing.

Methodology: AnyBokeh

The authors propose AnyBokeh, a physics-guided framework for any-to-any bokeh editing. Instead of treating source blur as a degradation to be removed, the method estimates the source blur state and transfers its optical characteristics to a target setting.

1. Physics-Guided Optical Fingerprint Transfer

The framework relies on the thin-lens model, where the Circle of Confusion (CoC) is linearly related to the disparity difference between a scene point and the focal plane.

Optical Fingerprint ( $\kappa_{src}$ ): The authors define a source-specific slope $\kappa_{src}$ that maps disparity differences to image-space defocus. This slope absorbs global optical factors (aperture, focal length, sensor size, resolution).
Estimation: The system jointly predicts a signed source CoC map ( $CoC_{src}$ ) and a disparity map ( $D$ ) from the input image.
Transfer: By modeling the linear relation $CoC \approx \kappa \cdot (D_{focus} - D)$ , the method estimates $\kappa_{src}$ from the predicted maps. This fingerprint is then analytically transferred to the desired target focus and aperture settings to compute the target CoC map ( $CoC_{tgt}$ ), avoiding the need for AIF reconstruction or test-time calibration.

2. Dual-CoC Conditioning

The editing process is handled by a generative editor (based on FLUX.1-Fill) conditioned on three inputs:

The source image ( $I_{src}$ ).
The source CoC map ( $CoC_{src}$ ).
The target CoC map ( $CoC_{tgt}$ ).

This dual-CoC conditioning allows the model to explicitly observe both the current blur state and the desired output state. This enables relative blur synthesis, where the editor performs spatially adaptive deblurring, preservation, or defocus rendering based on the transition between the two CoC maps.

3. UnrealBokeh Dataset

To train the CoC and disparity estimators under accurate physical supervision, the authors constructed UnrealBokeh, a high-fidelity synthetic dataset rendered in Unreal Engine.

It contains 11,477 images and 382,888 editing pairs.
It provides ground-truth depth, focus distance, aperture, focal length, sensor size, and image resolution for every image.
This allows for the computation of dense, signed ground-truth CoC maps, which are essential for training the Stage 1 estimator.

Key Contributions

CoC-Mediated Optical Fingerprint Transfer: A framework that estimates a source-specific optical fingerprint ( $\kappa_{src}$ ) from the observed blur and disparity, analytically transferring it to target settings. This eliminates the need for AIF reconstruction and test-time bokeh-level calibration.
Dual-CoC Conditioning: A novel conditioning strategy where the generative editor receives both source and target CoC maps. This turns any-to-any editing into a relative blur synthesis task, enabling precise spatial control over deblurring and defocus.
UnrealBokeh Dataset: A high-fidelity synthetic dataset with complete optical metadata and ground-truth CoC maps, enabling robust learning of source blur and geometry estimation.

Experimental Results

The authors evaluated AnyBokeh on real-world benchmarks (EBB! and RealBokeh) across three settings:

Any-to-Any Bokeh Editing: Editing images that already contain defocus. AnyBokeh outperformed "deblur-then-render" baselines (combining AIF estimators like Restormer/DRBNet with bokeh renderers like BokehMe/BokehDiff) in perceptual similarity (LPIPS, DISTS) and distribution realism (FID). Crucially, it achieved this without per-image calibration.
AIF-to-Bokeh Rendering: Even when the input is all-in-focus, AnyBokeh outperformed specialized rendering methods, demonstrating that the relative transfer formulation is robust even when source blur is weak.
Defocus Deblurring (Bokeh-to-AIF): In recovering sharp images from defocused inputs, AnyBokeh achieved superior DISTS and FID scores compared to restoration baselines, indicating better perceptual structure and realism. While LPIPS scores were slightly lower (attributed to the generative nature of recovering missing high-frequency details), visual inspection showed sharper boundaries and more realistic textures.

Significance and Limitations

Significance:
The paper argues that source blur is not merely noise but a carrier of useful optical information. By explicitly modeling and transferring the "optical fingerprint," AnyBokeh enables faithful and controllable editing across any-to-any scenarios without the artifacts introduced by forced AIF reconstruction. It bridges the gap between physical optics and generative editing, allowing for blind post-capture DoF manipulation.

Limitations:

Extreme Blur: When the input is severely defocused, high-frequency details are irretrievably lost. While AnyBokeh performs better than baselines, it cannot fully recover sharp details in these extreme cases.
Extreme Bokeh Changes: The method may struggle with extreme changes in bokeh ball size (e.g., reducing very large bokeh balls significantly), potentially leaving residual artifacts or hollow structures due to occlusion and limited training data coverage of such extremes.
Inference Efficiency: High-resolution inference relies on a tiling-and-stitching strategy to maintain aspect ratios, which increases runtime compared to single-pass inference.