Technical Summary: FOCUS – Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization

1. Problem Definition and Motivation

In-Context Object Localization (ICOL) is the task of identifying and localizing a specific target object in a query image based solely on a small set of visual support examples (images with bounding boxes) provided at inference time, without any parameter updates or training on the specific target.

While Vision-Language Models (VLMs) have advanced significantly, they struggle with ICOL in realistic, category-agnostic settings. Existing methods, such as IPLoc, often rely on semantic priors (e.g., category names or pseudo-labels) rather than pure visual evidence. This leads to two critical failure modes:

1. Category Bias: Models tend to localize the most prototypical or visually salient instance of a category rather than the specific instance defined by the support example (e.g., localizing any "bowl" instead of the specific "bowl" shown in the support).
2. Weak Visual Grounding: Attention mechanisms in current models often distribute focus diffusely across the query image or rely heavily on category tokens, failing to concentrate on the geometric and spatial cues provided by the support bounding boxes.

The paper argues that for applications like personalized visual search and instance-specific image editing, models must infer the target based on visual correspondence and spatial reasoning alone, completely decoupled from textual category labels.

2. Methodology: The FOCUS Framework

The authors propose FOCUS, a two-stage training framework designed to enforce visual grounding and eliminate category bias. The approach operates on a category-agnostic premise: the model receives only visual inputs (support images and their bounding boxes) and the query image, with no category names in the prompt.

Stage 1: Attention-Based Bounding Box Optimization (Supervised Fine-Tuning)

The first stage addresses the "weak attention" problem by explicitly optimizing the model's latent attention maps to prioritize spatial annotations.

• Prompt Specification: The input is an interleaved sequence of a natural language instruction, support images with their bounding boxes (serialized as text tokens), and the query image. Crucially, no category names are included.
• Attention Loss ($L_{bbox}$): The authors introduce a loss function that forces the attention weights from the query image tokens to concentrate on the bounding box annotation tokens of the support images.
  • A binary mask distinguishes between BBOX tokens and non-BBOX tokens.
  • The loss minimizes a margin-based hinge loss, encouraging the model to assign higher attention mass to BBOX tokens ($p^+$) than to non-BBOX tokens ($p^-$).
  • This ensures the model learns to "look" at the geometric constraints provided by the support examples rather than relying on semantic shortcuts.
• Implementation: This stage utilizes Supervised Fine-Tuning (SFT) with LoRA (Low-Rank Adaptation) on a base VLM (e.g., Qwen2-VL-7B).

Stage 2: Reinforcement Learning with Group Relative Policy Optimization (GRPO)

The second stage refines the precision of the predicted bounding boxes.

• Objective: To minimize the localization error (IoU) and ensure syntactic correctness of the output.
• Reward Function: A combined reward $R$ is used:
  1. IoU Reward ($r_{iou}$): The Intersection-over-Union between the predicted box and the ground truth.
  2. Formatting Reward ($r_{fmt}$): A binary reward ensuring the output follows the required format (e.g., <answer>[x_min, y_min, x_max, y_max]</answer>).
• Optimization: GRPO is employed to update the policy. Unlike PPO, GRPO does not require a learned critic; instead, it computes advantages by comparing sampled outputs within a group relative to the group's mean reward. This stabilizes training in few-shot settings and directly optimizes for high-quality alignment.

3. Key Contributions

1. Category-Agnostic Framework: The paper introduces a pure visual in-context localization setting that removes all category names from prompts, forcing the model to rely exclusively on visual appearance and geometry.
2. Attention Map Optimization: A novel loss function is proposed to explicitly align the model's attention with support bounding box tokens, correcting the tendency of VLMs to ignore spatial constraints in favor of semantic priors.
3. GRPO-Based Refinement: The integration of Group Relative Policy Optimization allows for direct minimization of localization error (IoU) and enforces strict output formatting, improving instance-level precision.
4. Efficiency and Scalability: The authors demonstrate that a 7B-parameter model trained with these objectives outperforms significantly larger models (up to 72B parameters), proving that context-aware localization objectives are more critical than mere model scaling.

4. Experimental Results

The authors evaluated FOCUS on a diverse set of benchmarks, including TAO, GOT-10k, LaSOT, PerSeg, and PerMIRS.

• Performance vs. Baselines: FOCUS (based on Qwen2-VL-7B) achieved state-of-the-art results, significantly outperforming strong baselines like IPLoc (7B), Qwen2-VL-72B, and InternVL2-76B.
  • On GOT-10k (zero-shot category generalization), FOCUS achieved 82.6% (4-shot), surpassing the 72B parameter Qwen2-VL model (59.9%) by a large margin.
  • On TAO (open-world, multi-object), FOCUS reached 68.5% (4-shot), compared to 55.6% for Qwen2-VL-72B.
• Generalization: The model showed robust generalization to unseen categories and domain shifts (e.g., synthetic data in PerSeg), maintaining high performance where category-dependent models failed to disambiguate instances.
• Ablation Studies:
  • Removing the attention loss resulted in a significant performance drop, confirming the necessity of forcing attention to BBOX tokens.
  • Adding GRPO on top of SFT + Attention Loss provided an additional 11.4% gain in the 4-shot setting on TAO.
  • Applying the attention loss to the first five layers of the transformer was found to be most effective for grounding, suggesting early layers are crucial for establishing spatial correspondence.

5. Significance and Claims

The paper claims that FOCUS successfully reframes in-context localization as a visual reasoning task rather than a semantic retrieval task. By eliminating category supervision and enforcing strict visual correspondence through attention constraints and policy optimization, the method achieves robust instance-level grounding.

The authors emphasize that their approach demonstrates that specialized training objectives can yield superior performance compared to simply scaling model size. A 7B model with FOCUS outperforms 72B models, highlighting that "context-aware localization objectives" are a more efficient path to solving the problem of category-agnostic object localization than relying on massive parameter counts alone. The work suggests that for user-driven, instance-specific applications, models must be trained to ignore semantic priors and focus strictly on visual evidence.