💻 computer science

Generative Refocusing: Flexible Defocus Control from a Single Image

This paper introduces Generative Refocusing, a two-step framework that combines DeblurNet and BokehNet to enable flexible, high-quality defocus control and custom aperture shapes from a single image by effectively merging synthetic and real-world bokeh data.

Original authors: Chun-Wei Tuan Mu, Cheng-De Fan, Jia-Bin Huang, Yu-Lun Liu

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chun-Wei Tuan Mu, Cheng-De Fan, Jia-Bin Huang, Yu-Lun Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you take a beautiful photo of a friend at a party. But when you look at it later, you realize you focused on the wrong person, or the background is too sharp and distracting. In the old days, you'd have to take the photo all over again, hoping to get it right. If you didn't have the right camera equipment, you might never get that perfect "blurry background" look (called bokeh) or fix the focus.

This paper introduces a new AI tool called Generative Refocusing (or GenRefocus). Think of it as a "Time Machine for Your Photos" that lets you change the focus and the blur after you've already taken the picture, even if you only have one single image.

Here is how it works, broken down into simple concepts:

1. The Two-Step "Magic Trick"

The authors realized that trying to fix a blurry photo and add a new blur at the same time is like trying to untangle a knot while simultaneously tying a new one. It's too hard. So, they split the job into two distinct steps, handled by two different AI "specialists":

Step 1: The "Sharpener" (DeblurNet)
Imagine your photo is a muddy, blurry painting. The first AI specialist is like a master restorer. Its only job is to look at that muddy mess and imagine what the scene would look like if it were perfectly sharp and in focus. It doesn't care about the blur yet; it just wants to recover the hidden details (like text on a sign or the texture of a face) and create a "perfectly clear" version of your photo in its mind.
Step 2: The "Artist" (BokehNet)
Once the "Sharpener" has created that perfect, clear version, the second specialist takes over. This is the artist. You tell the artist, "I want the focus here, and I want the background to be super blurry," or even, "I want the blurry lights in the background to look like hearts!" The artist then takes that clear image and paints the blur exactly where you want it, creating a realistic, dreamy effect.

2. Why Is This Better Than Previous Attempts?

Previous methods were like trying to drive a car with a broken steering wheel. They had three big problems:

They needed a "perfect" starting point: Most tools only worked if you gave them a photo that was already perfectly sharp. If your photo was blurry to begin with, they gave up. GenRefocus can fix a blurry photo first, then add the blur.
They were stuck in "Simulation Land": Many tools were trained on computer-generated images (like video game graphics). They looked okay, but they didn't feel real. They couldn't capture the messy, beautiful imperfections of real camera lenses.
They were boring: You could usually only change how much blur there was, not what shape the blur took.

GenRefocus fixes this by:

Mixing Real and Fake Data: The team trained their AI on a mix of computer simulations (to learn the rules of geometry) and thousands of real photos taken by real people. This taught the AI how real lenses actually behave, including their unique quirks.
Creative Control: You can now tell the AI to make the background lights look like stars, hearts, or triangles. It's like changing the shape of the aperture (the hole in the lens) without ever touching the camera.

3. The "Virtual Camera"

The paper describes this system as turning your single photo into a Virtual Camera.

Focus Plane Control: You can click on the background, and the AI will shift the focus there, making the background sharp and the foreground blurry.
Aperture Size Control: You can slide a bar to go from "everything is sharp" (like a landscape photo) to "super blurry background" (like a portrait).
Aperture Shape Control: You can upload a shape (like a heart), and the AI will make the out-of-focus lights in the background take that shape.

4. A Real-World Example

Imagine you took a photo of a street sign at night, but you accidentally focused on a car in the distance, so the sign is a blurry mess.

Old Way: You delete the photo.
GenRefocus Way:
1. You feed the blurry photo into the system.
2. The Sharpener guesses what the sign says and reconstructs the letters clearly.
3. You tell the Artist, "Focus on the sign, and make the streetlights in the background look like glowing hearts."
4. The system outputs a new photo where the sign is crisp and the background lights are glowing hearts.

The Catch (Limitations)

Like any magic, it has limits.

Transparent Objects: If you are looking through a glass window, the AI might get confused about what is in front and what is behind, leading to weird blurring.
Hallucinations: If the original photo is extremely blurry (like a smear), the AI might "guess" details that weren't there (like inventing a face where there was none). It's great for fixing mistakes, but not for reading microscopic text from a blurry snapshot.

Summary

Generative Refocusing is a powerful new tool that acts like a digital darkroom on steroids. It takes a single, imperfect photo, reconstructs the hidden sharp details, and then lets you paint the focus and blur exactly how you want, even letting you change the shape of the background lights. It turns a static photo into a flexible, creative canvas.

1. Problem Statement

Single-image refocusing aims to adjust the focus plane and depth of field (DoF) after an image has been captured. While essential for artistic photography, current methods face three critical limitations:

Input Rigidity: Most existing approaches require an all-in-focus (AIF) input or accurate depth maps, making them ineffective for images that are already defocused or blurry.
Data Limitations: Methods relying on synthetic data often fail to capture realistic optical aberrations. Conversely, real-world datasets are often restricted to isolated 1D variations (e.g., only aperture sweeps or only focus sweeps), lacking the joint distribution needed for flexible, dual-parameter control.
Limited Control: Current systems typically control only the size of the aperture (bokeh intensity) but not the shape (e.g., heart-shaped bokeh), limiting creative possibilities.

2. Methodology: Generative Refocusing (GenRefocus)

The authors propose a two-stage diffusion-based framework that decouples the task into Defocus Deblurring and Bokeh Synthesis. This architecture allows the system to accept any input image (blurry or sharp) and apply precise, user-defined controls.

A. Architecture Overview

The pipeline consists of two distinct modules based on the FLUX.1-dev diffusion transformer backbone:

DeblurNet (Stage 1):
- Goal: Recover a high-quality, all-in-focus (AIF) image from a diverse set of inputs (including heavily defocused images).
- Mechanism: It uses a unified token sequence concatenating noisy latent tokens ( $X_t$ ) and encoded input image tokens ( $\mathcal{E}(I_{in})$ ). It iteratively denoises to reconstruct the sharp AIF image ( $I_{aif}$ ).
- Training: Supervised using real paired data (defocused vs. AIF) from datasets like DPDD and RealBokeh.
BokehNet (Stage 2):
- Goal: Synthesize realistic bokeh effects on the recovered AIF image based on user controls.
- Inputs: The AIF image ( $I_{aif}$ ), a user-specified Focus Plane ( $D_{focus}$ ), a Bokeh Level ( $K$ ), and an optional Aperture Shape ( $A$ ).
- Defocus Map Generation: A monocular depth estimator ( $D$ ) is used to compute a defocus map: $D_{def} = K \cdot |D - D_{focus}|$ .
- Mechanism: Similar to DeblurNet, it concatenates condition tokens ( $I_{aif}$ , $D_{def}$ ) with noisy latents to iteratively denoise and synthesize the final refocused output ( $I_{out}$ ).

B. Novel Training Strategy (Hybrid Data Approach)

To overcome the lack of perfectly paired real-world data for joint focus/aperture control, the authors introduce a hybrid training scheme:

Synthetic Data: Used for pre-training to ensure geometric consistency. A simulator renders bokeh based on random focus planes and bokeh levels.
Real Data (ITW & LFDOF/RealBokeh):
- ITW Dataset: For real bokeh images, the system uses DeblurNet to generate a pseudo-AIF image. It estimates the focus plane using an in-focus mask (BiRefNet) and calculates the bokeh level ( $K$ ) using EXIF metadata (focal length, f-number) and sensor ratios.
- LFDOF/RealBokeh: For datasets lacking EXIF data, the authors introduce a simulator-in-the-loop calibration. They sweep through possible $K$ values, render the result, and select the $K$ that maximizes SSIM with the real target image. This creates reliable pseudo-labels for training.
Aperture Shape Control: A specialized PointLight-1K dataset was synthesized (using FLUX and DeblurNet) to teach the model how to render specific aperture shapes (e.g., hearts, stars) by conditioning on shape kernels.

C. Implementation Details

Backbone: FLUX.1-dev with LoRA (Low-Rank Adaptation).
Resolution Handling: Uses a tiling strategy to process high-resolution images (e.g., 4K) without losing fidelity or requiring resizing.
Inference: 28 denoising steps per stage.

3. Key Contributions

Flexible Input Pipeline: The first system capable of accepting images in any focus state (blurry or sharp) and converting them into a controllable virtual camera.
Hybrid Training Scheme: A novel method to complete missing control signals (defocus maps, focus planes, $K$ values) in real-world datasets, successfully unifying the geometric consistency of synthetic data with the authentic optical characteristics of real lenses.
Comprehensive Control: Supports control over Focus Plane, Bokeh Intensity, and Aperture Shape (a feature rarely supported in learning-based methods).
Two-Stage Decomposition: Proves that separating deblurring from bokeh synthesis yields superior results compared to direct end-to-end refocusing, as it prevents error propagation from depth estimation on blurry inputs.

4. Experimental Results

The method was evaluated on three tasks: Defocus Deblurring, Bokeh Synthesis, and Refocusing.

Defocus Deblurring: Outperformed state-of-the-art (SOTA) methods (e.g., Bokehlicious, DiffCamera) on RealDOF and DPDD benchmarks. It achieved the best scores in perceptual metrics (CLIP-IQA, MANIQA, MUSIQ) and successfully recovered text and fine details where others failed or introduced artifacts.
Bokeh Synthesis: On the new LF-Bokeh dataset, GenRefocus achieved the highest fidelity (lowest LPIPS/DISTS) and strong controllability (high LVCorr), surpassing diffusion baselines like BokehDiff and DiffCamera.
Refocusing: On the LF-Refocus dataset, it achieved SOTA performance in both fidelity and perceptual quality, correctly placing the focal plane and synthesizing natural bokeh, whereas competitors often produced spatially ambiguous blur or incorrect focus placement.
Ablation Studies:
- Confirmed that the two-stage pipeline significantly outperforms direct refocusing.
- Demonstrated that combining Synthetic, ITW, and LFDOF/RealBokeh data yields the best optical characteristics.

5. Significance and Impact

Democratizing Professional Photography: Enables users to achieve professional depth-of-field effects and correct focus errors from a single snapshot without specialized hardware (like light-field cameras or dual-pixel sensors).
Creative Freedom: The ability to change aperture shapes (e.g., creating heart-shaped bokeh) opens new avenues for artistic expression previously limited to physical lens modifications.
Robustness: By leveraging real-world data and a hybrid training approach, the model generalizes better to "in-the-wild" images, including low-light, grayscale, and severely defocused scenarios, outperforming methods reliant solely on synthetic simulators.
Limitations: The method relies on depth estimation; it may struggle with transparent surfaces (where depth is ambiguous) or extremely severe blur where hallucination of details can occur.

In conclusion, Generative Refocusing represents a significant leap forward in computational photography, offering a flexible, high-fidelity, and creative solution for post-capture focus control.