💻 computer science

Localize, Don't Beautify: Client-Side Control of Image-Editing APIs for Cosmetic Surgery Previews

This study demonstrates that client-side masking and compositing techniques can effectively localize cosmetic surgery edits to specific facial regions across various commercial APIs without requiring access to model internals, offering a low-cost solution to prevent unintended alterations to other facial features.

Original authors: Sukhrobbek Ilyosbekov

Published 2026-08-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sukhrobbek Ilyosbekov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to use a magical, all-knowing photo editor that lives on a distant server. You send it a picture of your face and a simple note: "Please make my jawline sharper." But this magic box is a bit of a show-off; it doesn't just sharpen your jaw. It also smooths your skin, brightens your teeth, changes the lighting, and maybe even gives you a different nose. It's like asking a chef to add a pinch of salt to your soup, and they decide to rewrite the entire recipe, changing the vegetables, the broth, and the garnish, too. This is the world of "black-box" image editing APIs: powerful tools where you can see the input and the output, but you have no idea how the magic happens inside, and you can't tell the tool to "only touch this one spot."

For decades, surgeons have used these kinds of previews to show patients what a cosmetic procedure might look like. But if the preview changes too much of the face, the patient might not recognize themselves, or they might get excited about changes they didn't ask for. The big question for scientists is: If you can't get inside the tool to tell it to be careful, can you, the person using it, force it to stay in its lane? This paper explores a clever trick to solve that problem without needing to break into the tool's secret code.

The researchers set up a little experiment to see if they could stop these image editors from wandering off the path. They treated six different commercial photo-editing tools like a group of eager but slightly overzealous assistants. They gave them a simple job: edit a specific part of a face (like a nose or a jaw) while leaving the rest of the face exactly as it was. First, they tried just sending a text prompt, like saying, "Fix the nose, but don't touch anything else." As they suspected, the tools ignored the "don't touch" part and changed the whole face, smoothing skin and altering lighting in the process.

Then, the researchers tried a different approach, which they call a "masked composite." Imagine you ask a painter to fix a smudge on a window, but you know they are going to paint the whole wall. Instead of arguing with the painter, you wait until they finish, take a piece of tape with a hole cut in it (a mask), and stick it over the window. You then carefully cut out the painted window from the wall and paste it back onto your original photo, while keeping the rest of the wall exactly as it was before. In the digital world, the researchers did this by taking the edited photo, cutting out only the part they wanted to change, and pasting it back onto the original photo using a soft, feathered edge so it looked natural.

The results were surprisingly clear. When the researchers used just the text prompt, the editors changed a lot of the face they weren't supposed to touch. But when they used the "cut and paste" trick, they successfully kept the rest of the face untouched. In fact, for the faces they could measure, this method improved how well the edit stayed in the right spot by a huge margin (a median improvement of 0.446 on their scale), while still making the requested change to the nose or jaw. The "cut and paste" method worked for every single editor they tested, from the most expensive ones to the cheaper ones.

However, the paper also found some important limits. One of the editors they tested had a special "inpainting" mode, which is supposed to let the tool know exactly where to paint. But even that special mode didn't do a better job at keeping the changes local than the simple "cut and paste" trick the researchers invented. Furthermore, while the researchers could measure that the faces looked more like the original person after using their trick, they also found that none of the editors actually made the face look more like the real "after" surgery photo. In other words, the tools didn't get the surgery right; they just got better at not messing up the rest of the face.

The author is very careful to say this isn't a magic cure-all for cosmetic surgery. They didn't have real surgeons look at the pictures to see if the changes looked medically realistic. They also noted that for some angles, like a side profile, their "cut and paste" trick didn't work as well because the computer couldn't find the face landmarks to make the mask. But the main takeaway is a reassuring one: even if you are stuck using a black-box tool that you can't control from the inside, you can still force it to stay in its lane by using a simple "cut and paste" trick on your own computer. It's a low-cost, easy way to make sure your digital preview doesn't turn your face into someone else's.

Technical Summary: Localize, Don't Beautify

Problem Statement
Commercial image-editing APIs, which function as black-box models accessible only via public endpoints, frequently fail to confine edits to specific regions when requested for cosmetic surgery previews. A request to alter a specific feature (e.g., a nose reduction or jawline repositioning) often triggers "off-target" changes, such as skin smoothing, lighting adjustments, or identity shifts across the entire face. Existing methods for localizing edits—such as Blended Diffusion, SDEdit, or attention injection—require access to the model's internal states (latents, attention maps, or sampling loops), which public APIs do not expose. Consequently, there is a gap in understanding how much control can be exerted from the client side alone to prevent these unwanted global modifications without access to the model's internals.

Methodology
The author conducted a pilot benchmark evaluating six commercial editing configurations (GPT Image 2 at two tiers, Nano Banana Pro/2, Seedream 5.0 Lite, FLUX.2 [pro]) and one mask-based inpainting model (Qwen-Image-Edit). The study utilized a "control ladder" approach, testing three levels of client-side intervention on 210 attempted edits across 15 distinct faces (12 frontal facelift and 14 rhinoplasty inputs, with some overlap in analysis):

Prompt Only: The model receives the image and text instruction directly. This serves as the baseline for uncontrolled output.
Masked Composite: The model edits the full image, but the client-side system aligns the output to the input using landmark-derived similarity warps. The edited region is then extracted via a feathered mask (derived from MediaPipe face meshes or InsightFace landmarks) and composited back onto the original photograph. Pixels outside the mask are strictly preserved from the input.
Masked Inpaint: For models supporting it, the client sends the region mask explicitly, asking the model to inpaint only within that area.

Diagnostic Protocol
The evaluation employed a two-part diagnostic system, explicitly noting that these are systems diagnostics rather than clinical validation tools:

Identity Preservation: Measured using ArcFace cosine similarity between the input and output faces. A threshold of 0.6 was used as a heuristic reference.
Regional Localization: Measured using the CIELAB $\Delta E_{76}$ color difference. The metric calculates the ratio of pixel change in the target region ( $\Delta E_{tgt}$ ) versus a disjoint "keep zone" (upper face) ( $\Delta E_{off}$ ). The localization score is defined as $\frac{\Delta E_{tgt}}{\Delta E_{tgt} + \Delta E_{off}}$ .
Postoperative Reference: For faces with paired postoperative photographs, the study compared the ArcFace embedding of the edit against the postoperative photo, using the unedited input's similarity to the postoperative photo as a baseline.

Key Results

Localization: Masked compositing significantly improved localization. The median localization ratio for prompt-only outputs was 0.538, whereas the masked composite achieved a median of 0.985. Across 12 frontal faces, the composite improved localization by a median of 0.446 (95% bootstrap interval 0.421–0.457) compared to prompt-only outputs, while maintaining similar levels of change in the target region.
Identity Retention: Editors varied widely in their trade-off between edit strength and identity retention. FLUX.2 [pro] produced the strongest edits but the lowest identity scores (median 0.629 for prompt-only, dropping to 0.195 in some cases). Nano Banana configurations and Seedream 5.0 Lite retained identity better (medians >0.90) but showed less aggressive target changes.
Inpainting Performance: The tested Qwen-Image-Edit model did not outperform the simple client-side composite. On rhinoplasty tasks, it produced near-copy outputs with low target change, and on facelift tasks, it showed lower identity retention than the composite approach.
Clinical Accuracy: No editor moved the output embeddings systematically closer to the postoperative photographs. The median change in embedding space was negative ($-0.029$), indicating that, in terms of identity-embedding similarity, the edits moved slightly further from the observed surgical outcome rather than closer.

Significance and Claims
The paper claims to be the first controlled comparison of client-side region confinement across multiple commercial, black-box editing APIs for cosmetic surgery previews. Its primary contribution is demonstrating that region confinement can be enforced entirely from the client side without access to model internals.

The author explicitly states that this is a study of control, not clinical accuracy. They do not claim that the generated previews are anatomically correct, clinically plausible, or safe for patient counseling. No surgeons rated the outputs, and the study lacks demographic diversity analysis. The significance lies in the finding that a simple post-hoc compositing strategy (mask, alignment, and blend) effectively prevents off-target pixel changes across diverse black-box models at a low provider cost (median $0.045 per image), whereas prompt engineering alone fails to protect facial zones. The author concludes that while this method solves the localization problem for black-box APIs, future work requires blinded surgeon ratings, larger cohorts, and anatomy-aware metrics to validate clinical utility.

Technical Summary: Localize, Don't Beautify

More like this