💬 NLP

FoR-SALE: Frame of Reference-guided Spatial Adjustment in LLM-based Diffusion Editing

The paper introduces FoR-SALE, a novel framework that enhances text-to-image editing by evaluating and correcting spatial misalignments between text descriptions and generated images through frame-of-reference-guided perspective mapping and latent-space adjustments, significantly improving state-of-the-art model performance on spatial understanding benchmarks.

Original authors: Tanawan Premsri, Parisa Kordjamshidi

Published 2026-08-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tanawan Premsri, Parisa Kordjamshidi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to describe a scene to a friend who is drawing it for you. If you say, "The cat is to the left of the dog," your friend probably draws the cat on the left side of the paper. That's easy because you are both looking at the scene from the same angle: your own eyes. But what if you say, "The cat is to the left of the dog, from the dog's point of view"? Suddenly, the drawing gets tricky. If the dog is facing right, its "left" is actually on the right side of your paper. If the dog is facing away from you, its "left" is on your left. This mental gymnastics is called understanding the "Frame of Reference." It's the difference between describing the world from your camera's eye versus from the perspective of the objects themselves.

For a long time, computers have been great at following simple instructions like "put the cat on the left." But when it comes to these tricky, object-centric perspectives, even the smartest artificial intelligence (AI) artists get confused. They tend to ignore the object's viewpoint and just draw everything from the camera's perspective, leading to messy, incorrect images. This paper tackles that specific confusion. It introduces a new system designed to act like a super-smart editor that doesn't just draw the picture, but also understands who is looking at what, and fixes the drawing if the perspective is wrong.

The Problem: AI's "Camera-Only" Blind Spot

Current AI models that turn text into images are incredibly talented, but they have a blind spot. They are like artists who only know how to paint from a single, fixed camera angle. If you ask them to draw a scene based on a description like "The ball is behind the chair from the chair's perspective," the AI often gets it wrong. It might put the ball behind the chair from your view, ignoring the fact that the chair might be facing a different way. The paper notes that even the most advanced models today struggle significantly with these "non-camera" perspectives, often failing to get the spatial relationships right.

The Solution: FoR-SALE (The Perspective Detective)

The authors propose a new framework called FoR-SALE (Frame of Reference-guided Spatial Adjustment in LLM-based Diffusion Editing). Think of FoR-SALE not as a new artist, but as a brilliant art critic and editor who steps in after the AI has made its first draft.

Here is how the process works, step-by-step:

The First Draft: The AI generates an image based on your text. Let's say you asked for a red chicken on the left of a chair, but from the chair's view. The AI draws a chicken, but maybe it's on the wrong side because it forgot to check which way the chair is facing.
The Detective Work: FoR-SALE uses a "Visual Perception Module" to look at the generated image. It acts like a scanner, identifying where the objects are, how deep they are, and most importantly, which way they are facing. It's like the editor putting on 3D glasses to see the orientation of every object.
The Translation: This is the magic part. The system uses a "FoR Interpreter" to translate your tricky instruction into a language the AI understands better. If you said, "From the chair's view, the chicken is on the left," and the chair is facing right, the interpreter translates this to: "From the camera's view, the chicken is actually on the right." It converts the object's perspective into the camera's perspective so the computer doesn't get lost.
The Fix: Once the system knows what the image should look like, it uses special "latent-space operations" to edit the picture. It doesn't just erase and redraw; it performs precise surgical adjustments. It can change an object's facing direction (making the chair turn around) or adjust its depth (moving it closer or further away) to match the corrected plan.

What They Found: A Significant Improvement

The researchers tested this system on several benchmarks designed to trick AI with spatial puzzles. They found that FoR-SALE works remarkably well, especially for the hardest cases where the perspective comes from an object rather than a camera.

The Numbers: When they applied just one round of correction, the system improved the accuracy of state-of-the-art image generators by up to 7.0 percentage points. If they let the system run for three rounds of corrections, the improvement jumped to 13.1 percentage points.
The "Intrinsic" Challenge: The system shined brightest on "intrinsic" frames of reference (where the perspective belongs to an object). For example, with the GPT-4o model, FoR-SALE boosted the accuracy of intrinsic spatial descriptions from a low 24.35% up to 35.42% in a single round.
Generalization: The system didn't just work on simple two-object scenes. When tested on more complex scenes with multiple objects and varied language, it remained effective, suggesting it's a robust tool for understanding spatial relationships.

The Limits: It's Not Magic (Yet)

While FoR-SALE is a major step forward, the authors are careful to note it's not a perfect solution. The system still struggles with certain 3D aspects, particularly when it needs to change an object's depth or facing direction in a single step. Sometimes, the editing process can accidentally create extra objects or miss one entirely, though this happens less often than with previous methods. The paper suggests that while the "brain" (the reasoning part) is getting very good at understanding perspective, the "hands" (the image editing tools) still have some trouble executing those complex 3D changes perfectly.

In short, FoR-SALE teaches AI to stop assuming everyone sees the world from the same angle. By acting as a perspective translator and a precise editor, it helps AI generate images that truly respect the viewpoint of the objects within them, making the digital world a little more aligned with how we humans actually see it.

Technical Summary: FoR-SALE

Problem Statement

Current text-to-image (T2I) generation models, including state-of-the-art (SOTA) systems like GPT-4o and FLUX.1, exhibit a significant performance gap when interpreting spatial expressions derived from non-camera perspectives. While these models often succeed with camera-centric (relative) descriptions, they struggle to accurately render scenes where spatial relations are defined from an object's intrinsic perspective (e.g., "to the left of the chair from the chair's view"). This limitation stems from a lack of robust reasoning regarding the Frame of Reference (FoR), a concept well-studied in cognitive linguistics but under-addressed in Multimodal Large Language Models (MLLMs) and diffusion-based generation. Existing self-correcting frameworks, such as Self-correcting LLM-controlled Diffusion (SLD), improve spatial alignment but fail to account for FoR, limiting their ability to handle prompts grounded in non-camera perspectives.

Methodology: FoR-SALE

The authors propose FoR-SALE (Frame of Reference-guided Spatial Adjustment in LLM-based Diffusion Editing), an extension of the SLD pipeline designed to explicitly model and correct FoR misalignments. The framework operates through a multi-stage pipeline:

Initial Generation & Perception:
- An input text prompt ( $T_1$ ) is passed to a T2I model to generate an initial image ( $I_1$ ).
- An LLM Parser extracts key objects and attributes from $T_1$ .
- A Visual Perception Module analyzes $I_1$ to extract visual properties: object bounding boxes, depth maps, and orientation angles. This module utilizes open-vocabulary object detection (OWLv2), depth estimation (DPT), and orientation detection (OrientAnything) to produce a structured visual list ($VL$) containing object attributes, coordinates, depth, and facing directions.
FoR Interpretation & Layout Revision:
- FoR Interpreter: This novel LLM component takes the original text ( $T_1$ ) and the visual state ($VL$) as input. Its primary function is to identify the FoR in the prompt and convert spatial expressions from the object-centric perspective to a unified camera perspective. It employs a set of 32 manually defined rules (covering 8 facing directions and 4 spatial relations) to perform this conversion, outputting a rewritten prompt ( $T'$ ).
- Layout Interpreter: A second LLM analyzes $T'$ against the current visual state ($VL$) to determine alignment. If misalignment is detected, it proposes a revised layout ( $\tilde{VL}$ ) that satisfies the spatial constraints under the camera perspective.
Latent-Space Editing:
- The system compares the initial layout ($VL$) with the revised layout ( $\tilde{VL}$ ) to identify specific discrepancies.
- Based on these discrepancies, a sequence of editing operations is generated and applied in the latent space of the diffusion model (using Stable Diffusion 1.5 as the base).
- Novel Operations: In addition to standard operations (Addition, Deletion, Reposition, Attribute Modification), FoR-SALE introduces two FoR-specific operations:
  - Facing Direction Modification: Uses segmentation and diffusion to alter an object's orientation to match the required facing direction.
  - Depth Modification: Synthesizes a new depth map for an object based on target depth values and uses ControlNet to regenerate the object with the specified depth.
Iteration: The process can be repeated for multiple rounds of correction to further refine the image.

Key Contributions

First FoR-Aware Self-Correcting Framework: The paper introduces the first framework that explicitly incorporates Frame of Reference reasoning into the text-to-image generation and editing pipeline.
Novel Latent Editing Operations: The authors develop specific latent-space operations for modifying object facing directions and depth, which are critical for resolving FoR-based spatial errors.
Enhanced Benchmarking: The work augments existing benchmarks (LMD and FoREST) to create FoR-LMD and FoREST-G. These datasets include diagnostic subsets covering multi-object scenes, expanded relation types (diagonal/vertical), and naturalized prompts to rigorously test generalization beyond simple two-object settings.

Results

The framework was evaluated on three benchmarks (FoR-LMD, FoREST, and FoREST-G) using various base T2I models, including FLUX.1, Nano-Banana-Pro, and GPT-4o.

Performance Gains: FoR-SALE improves the spatial alignment of SOTA T2I models by up to 7.0 percentage points (pp) in a single correction round and up to 13.1 pp after three rounds.
Intrinsic FoR Handling: The framework demonstrates particular efficacy in intrinsic FoR cases, where baseline models typically fail. For instance, on the FoREST benchmark, FoR-SALE improved GPT-4o's intrinsic accuracy from 35.56% to 42.22% in three rounds, and significantly boosted FLUX.1's performance in similar settings.
Generalization: On the FoREST-G diagnostic subsets (Complex, Relation-Augmented, Language-Variation), FoR-SALE consistently achieved the highest intrinsic-FoR accuracy across all subsets, demonstrating robustness in complex scenes and with varied prompt phrasing.
Ablation Studies: Removing the FoR Interpreter or the new editing operations (Facing Direction and Depth Modification) resulted in significant performance drops, confirming the necessity of both components for effective FoR reasoning.

Significance and Claims

The paper claims that FoR-SALE addresses a fundamental limitation in current generative AI: the inability to reason about spatial relations from perspectives other than the camera's. By bridging the gap between cognitive linguistics (FoR) and diffusion editing, the framework enables more accurate and flexible text-to-image generation. The authors emphasize that while current diffusion models struggle with 3D spatial configurations (orientation and depth), FoR-SALE provides a modular approach to mitigate these issues through explicit reasoning and targeted latent-space adjustments. The work establishes a new baseline for FoR-aware spatial reasoning in T2I systems, demonstrating that explicit perspective conversion and specialized editing operations can significantly enhance the spatial fidelity of generated images.