ProCap: Projection-Aware Captioning for Spatial Augmented Reality

This paper introduces ProCap, a novel framework that decouples projected content from physical scenes in Spatial Augmented Reality through automated segmentation and region-aware retrieval, supported by the new RGBP benchmark dataset and a dual-captioning evaluation protocol to overcome virtual-physical ambiguity in Vision Language Models.

The Gist

Imagine you are sitting in a room with a wooden table. On that table, there is a real wooden cutting board and two blue vases. Suddenly, a projector shines a picture of a surfer riding a giant wave right onto the table.

To your eyes, it looks like a magical, messy mix: a surfer is "riding" the cutting board, and the blue vases are now part of the ocean.

Now, imagine you ask a very smart robot (an AI) to describe what it sees. If you ask a standard AI, it might get confused. It might say, "I see a wooden table with a real surfer and a blue vase." It thinks the surfer is a physical object sitting on the table, not a picture of light. This is the core problem this paper solves.

Here is a simple breakdown of the paper "ProCap: Projection-Aware Captioning for Spatial Augmented Reality":

1. The Problem: The "Magic Trick" Confusion

In Spatial Augmented Reality (SAR), we project digital images onto real-world objects to make them look cool or informative. But to a computer, a picture of a surfer projected onto a table looks just like a real surfer sitting on the table.

Current AI models (called Vision Language Models) are like people who have only ever looked at photos of real objects. When they see this "magic trick," they get confused. They can't tell the difference between:

• The Stage: The real physical world (the table, the vases).
• The Show: The projected light (the surfer, the wave).

They mix them up, leading to "hallucinations" where they describe things that aren't physically there.

2. The Solution: ProCap (The "Smart Stage Manager")

The authors created a new system called ProCap. Think of ProCap as a super-smart stage manager who knows exactly where the actors (real objects) are and where the holograms (projections) are.

ProCap works in two main steps:

• Step 1: The "Cut-Out" Trick (Segmentation)
  Imagine ProCap puts on special glasses that can see the "invisible" lines where the light starts and stops. It draws a digital outline around the projected image. It effectively says, "Okay, I'm going to ignore the surfer for a second and just look at the table." Then, it looks at the surfer separately. This stops the AI from thinking the surfer is a real, 3D object.

• Step 2: The "Cheat Sheet" (Retrieval)
  Projected images often look blurry, stretched, or weird because of the angle of the light or the shape of the table. It's like trying to read a newspaper held up to a curved mirror.
  ProCap doesn't just guess what the blurry image is. Instead, it has a massive Cheat Sheet (a database of clean, perfect images and names). When it sees a blurry, stretched picture of a car, it doesn't try to guess; it looks at its cheat sheet and says, "Ah, that's a distorted image of a Volkswagen Beetle." It grabs the correct name from its database to make sure the description is accurate.

3. The New "Textbook": The RGBP Dataset

To teach this new system, the researchers couldn't use old textbooks (existing AI datasets) because they only had pictures of real things. They had to write a brand new textbook called RGBP.

• What's in it? They took 65 different physical scenes (like a kitchen counter, a living room) and projected over 180,000 different images onto them.
• The Secret Sauce: For every single photo, they wrote two descriptions:
  1. One describing the real room (ignoring the projection).
  2. One describing the projection (ignoring the room).
     This teaches the AI to be bilingual: fluent in "Real World" and fluent in "Projected World."

4. Why Does This Matter?

Imagine a future where robots or smart assistants can walk into a factory or a museum and understand what's happening.

• Without ProCap: A robot sees a projection of a "Danger: Hot Surface" warning on a machine and thinks, "Oh, there is a real, glowing red sign made of glass on this machine." It might try to touch it or move it.
• With ProCap: The robot sees the machine and the warning separately. It understands, "The machine is metal, and there is a digital warning projected onto it. I should not touch the machine."

The Big Picture Analogy

Think of a Magic Show.

• The Audience (Old AI): Sees a rabbit appear out of a hat and thinks, "Wow, rabbits grow out of hats!" (They believe the illusion is real).
• The Magician (ProCap): Knows exactly how the trick works. They can tell you, "There is a real hat on the table, and there is a real rabbit hidden in a secret compartment, but the 'magic' is just a projection of light."

ProCap is the technology that finally lets computers understand the difference between the real world and the digital illusions we project onto it, allowing them to interact with our augmented reality world safely and intelligently.

Technical Summary

1. Problem Statement

Spatial Augmented Reality (SAR) projects digital content directly onto physical surfaces using projectors, creating immersive experiences without head-mounted displays. However, enabling intelligent interaction (e.g., reasoning, user queries) in SAR requires a system that can semantically distinguish between the physical scene and the projected content.

Current Vision Language Models (VLMs) fail in this domain due to three critical gaps:

1. Virtual-Physical Ambiguity: Standard VLMs assume all visual elements are physical objects. They often "merge" descriptions, misidentifying projected images (e.g., a surfer on a wave) as physical objects printed on a surface.
2. Perceptual Degradation: Projected content suffers from geometric and photometric distortions (due to surface curvature, lighting, and material), causing standard VLMs to generate hallucinated or low-fidelity captions.
3. Lack of Benchmarks: Existing datasets focus on low-level tasks (calibration, tracking) or natural images. There is no large-scale dataset with decoupled annotations (separate ground truth for the scene and the projection) to train or evaluate models on SAR-specific semantic understanding.

2. Methodology: The ProCap Framework

The authors propose ProCap, a novel framework that explicitly decouples the physical scene from the projected overlay using a two-stage pipeline.

A. Data Acquisition: The RGBP Dataset

To address the data gap, the authors introduced RGBP (RGB + Projections), the first large-scale SAR semantic benchmark.

• Scale: 65 diverse physical scenes and over 180,000 projections.
• Annotation: Features dense, decoupled ground truth (GT) including:
  • Coarse binary segmentation masks for projected regions.
  • Two separate captions per image: one for the Physical Scene and one for the Projected Content.
• Variability: Captured under varying lighting conditions (2.9–546 lux) and surface geometries (planar, mildly curved, highly curved).

B. Architecture Overview

ProCap treats the observed image $I$ as a composite $I = I_s \oplus I_p$ (Physical Scene $\oplus$ Projection). The framework operates as follows:

1. Feature Extraction & Projection Segmentation:

   • A frozen CLIP ViT-g encoder extracts coarse features from the input image.
   • A Segmentation Module (using deconvolutional layers and a convolutional network) generates a coarse binary mask ($I_m$) to isolate the projected region from the physical scene.
   • Mask Pooling is applied to extract projection-specific features ($Z_p$) while retaining global scene features ($Z_c$).

2. Region-Aware Retrieval & Captioning:

   • Dual Q-Formers: Two specialized query-based transformers process the features:
     • A Scene Q-Former processes $Z_c$ to generate scene embeddings ($Q_s$).
     • A Projection Q-Former processes $Z_p$ to generate projection embeddings ($Q_p$).
   • Semantic Retrieval: To combat distortion-induced hallucinations, the projection embeddings ($Q_p$) are used to retrieve top-K object names from an external knowledge base (derived from LVIS). A Knowledge Q-Former distills these retrieved names into a clean semantic context embedding ($Q_k$).
   • Fusion: The projection embedding is concatenated with the retrieved semantic context ($Q_p; Q_k$).

3. Dual-Captioning Generation:

   • The embeddings are projected into the input space of a frozen Large Language Model (LLM) decoder.
   • Task-specific tokens ([SCENE] and [PROJ]) are prepended to the generation sequence to guide the LLM in producing two distinct, independent captions simultaneously.

4. Training Objective:

   • The model is trained end-to-end using a weighted sum of three losses: Scene Captioning Loss, Projection Captioning Loss, and Projection Segmentation Loss (Binary Cross-Entropy).

3. Key Contributions

1. ProCap Framework: A two-stage pipeline that resolves virtual-physical ambiguity via automated segmentation and region-aware semantic retrieval, enabling robust dual-captioning.
2. RGBP Dataset: The first large-scale SAR dataset (180k+ images) with decoupled annotations, shifting the focus from low-level calibration to high-level semantic understanding.
3. Dual-Captioning Evaluation Protocol: A novel evaluation metric using task-specific tokens to independently assess a model's ability to describe the physical scene and the projection, preventing metric bias caused by context confusion.

4. Experimental Results

The authors evaluated ProCap against state-of-the-art VLMs (FastVLM, Qwen3-VL) on the RGBP benchmark.

• Performance on Seen Scenes:
  • Scene Captioning: ProCap variants significantly outperformed baselines. For instance, ProCap (TinyLlama-1.1B) achieved a CIDEr score of 70.27 on COCO, compared to ~2.38 for the best baseline (Qwen3-VL-8B).
  • Projection Captioning: Baselines struggled severely (max CIDEr ~11.56) due to hallucinations. ProCap variants achieved CIDEr scores of 78.99 (Vicuna-1.5-7B) and 76.98 (OpenLLaMA-3B).
• Generalization to Unseen Scenes: ProCap maintained a significant advantage in both tasks on 5 unseen physical scenes, demonstrating robustness to new environments and novel projection content.
• Ablation Studies:
  • Segmentation: Removing the mask pooling module degraded scene description quality (lower CIDEr) and caused projection descriptions to bleed into scene descriptions.
  • Retrieval: Removing the region-aware retrieval (using null tokens) caused a massive drop in projection CIDEr (from 86.26 to 67.98), confirming that external semantic knowledge is crucial for correcting distorted projections.
  • Dual-Task: Training a single model for both tasks proved more effective than training separate specialist models, as the framework successfully decoupled feature spaces without needing separate backbones.

5. Significance and Future Impact

• Enabling Intelligent SAR: ProCap provides the semantic foundation necessary for SAR systems to transition from static displays to autonomous, instruction-aware agents capable of reasoning about complex mixed-reality environments.
• MoE Integration: The authors suggest ProCap can serve as a specialized "expert" module within Mixture-of-Experts (MoE) architectures, allowing high-level routers to bypass generalist reasoning when SAR elements are detected, thereby reducing hallucinations.
• Generative Applications: The decoupled annotations in RGBP enable the training of generative models for SAR, allowing for natural language-driven synthesis of physical scenes and projected content independently.

In conclusion, ProCap effectively bridges the gap between computer vision and spatial augmented reality, solving the critical issue of virtual-physical ambiguity and establishing a new standard for evaluating and developing SAR-aware AI systems.