💻 computer science

Can Vision-Language Models Assess Proxemic Risk from Egocentric Robot Images?

This study evaluates open-source vision-language models for assessing proxemic risk from egocentric robot images, finding that while fine-tuning offers only modest overall gains, advanced prompting significantly improves high-danger detection in specific models like Qwen-VL, though correct safety classifications do not necessarily correlate with accurate spatial grounding.

Original authors: Vladyslava Rudas, Dmytro Kuzmenko

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Vladyslava Rudas, Dmytro Kuzmenko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to walk through a busy coffee shop. You don't just want it to see a table; you want it to feel the space around it. This is the world of proxemics, a fancy word for the study of how much personal space humans need to feel comfortable. Think of it like invisible bubbles: a tight bubble for a best friend, a medium one for a coworker, and a wide one for a stranger. If a robot pops into your tight bubble without asking, it's rude and potentially dangerous.

Now, imagine giving that robot a camera on its head (an "egocentric" view) so it sees the world exactly as we do. The big question scientists are asking is: Can we teach a robot to look at a picture and instantly know, "Oh no, I'm too close to that person!" or "Phew, I have plenty of room"? This is where Vision-Language Models (VLMs) come in. These are super-smart AI programs that can look at an image and talk about it, kind of like a robot that can read a picture book and tell you the story. Researchers are excited because if these AIs can understand personal space just by looking, we might not need expensive, heavy sensors on every robot. We could just give them a camera and a brain.

But here is the twist: just because an AI can describe a picture doesn't mean it truly understands where things are in 3D space. This paper puts three different AI "brains" to the test to see if they can act as a safety guard for robots.

The Experiment: Can AI Spot Danger?

The researchers built a special training set using 1,243 photos taken from a robot's point of view in real-world settings. They labeled every photo with a "danger level" based on how close people were to the robot:

High Danger: Someone is right in the robot's face (collision imminent!).
Moderate Danger: Someone is close, and the robot needs to move carefully.
Low Danger: Someone is nearby, but the robot just needs to keep an eye on them.
Minimum Danger: The coast is clear.

They tested three different open-source AI models: InternVL, SmolVLM, and Qwen-VL. They tried to teach these models in two ways: first, by just asking them questions in different styles (like giving a simple command vs. a complex, step-by-step reasoning puzzle), and second, by giving them a tiny bit of extra training (called "fine-tuning") on 200 specific images.

The Results: One Hero, Two Strugglers

The findings were a mix of "not bad" and "not good enough."

1. The "Random Guess" Problem
Without any special training, all three models performed almost exactly like a student guessing on a multiple-choice test. Their overall accuracy was barely better than flipping a coin. Even after the extra training, the general ability to sort all four danger levels correctly didn't improve much. It's as if the models were good at describing the coffee shop but terrible at knowing when to stop walking.

2. The Qwen Exception
However, there was one standout hero: Qwen-VL. While the other two models (InternVL and SmolVLM) basically failed to spot the most dangerous situations (missing almost all of them), Qwen-VL actually got better at spotting "High Danger" scenarios.

Crucially, this improvement came from a specific combination of how they asked the question and how they trained the model. The best results were achieved when using a very specific, complex prompt that asked the AI to "think step-by-step" along with a first round of fine-tuning. With this setup, Qwen-VL managed to catch about 79% of the high-danger situations.
The other models, even with the same complex prompts and training, still missed almost everything.
Interestingly, giving Qwen-VL additional training (a second round of fine-tuning) didn't help much more; the biggest boost came from that specific pairing of the advanced "thinking" prompt and the initial fine-tuning round.

3. The "Blind Spot" Surprise
Here is the most interesting part. The researchers checked if the AI was actually looking at the person to decide if it was dangerous. They asked the AI to draw a box around the person it was worried about.

The Shock: Even when Qwen-VL correctly shouted "DANGER!", it often didn't draw the box around the right person. Sometimes it pointed at the floor, sometimes at a wall, and sometimes at the right person.
The Conclusion: The paper suggests that the model might be guessing the danger level based on a "feeling" or a pattern in the whole image, rather than actually understanding the specific distance to the person. It's like a student who gets the right answer on a math test but can't show the work or point to the numbers they used.

What This Means for Robot Safety

The paper concludes that while these AI models are getting smarter, they aren't quite ready to be the sole safety guard for a robot walking through a crowd.

They can't be trusted yet: If you rely on them, you might miss a collision because the model thinks it's safe when it's not, or it might panic when it's safe.
Prompting and specific training matter: Giving the models a little extra training helped Qwen-VL only slightly on its own, but it didn't fix the other models. The biggest gains came from asking the model to reason carefully combined with a specific fine-tuning configuration.
The "Blind" Danger: The biggest takeaway is that getting the right label (High Danger) doesn't mean the robot knows why it's dangerous. It might be right for the wrong reasons.

In short, these Vision-Language Models are like very chatty tourists who can describe a scene beautifully but often get lost when asked to navigate it. They show promise, especially the Qwen model when asked to think carefully and trained just right, but they still need a lot more practice before we can let them drive a robot through a busy street without a human watching their back.

Technical Summary: Can Vision-Language Models Assess Proxemic Risk from Egocentric Robot Images?

Problem Statement
Safe embodied navigation for mobile robots in human environments requires the continuous assessment of proxemic risk—the potential for a robot to intrude upon personal space or cause collisions. While existing safety systems often rely on explicit geometric reasoning or specialized depth sensors, this study investigates whether universal Vision-Language Models (VLMs) can replace or complement these systems. The core challenge is to determine if VLMs, operating solely on egocentric (first-person) visual data, can accurately classify proxemic danger levels and provide the necessary spatial grounding (identifying where the danger lies) to inform robot control loops.

Methodology
The authors constructed a new dataset of 1,243 egocentric images derived from the JRDB dataset, categorized into four proxemic danger levels based on Edward T. Hall's zones: High, Moderate, Low, and Minimum. The dataset reflects real-world class imbalances, with high-danger scenarios being the least frequent but most critical.

The study evaluated three open-weight VLMs suitable for local deployment:

Qwen2.5-VL-3B-Instruct (3B parameters)
InternVL3.5-4B (4B parameters)
SmolVLM2-2.2B-Instruct (2.2B parameters)

The evaluation framework involved two primary experimental variables:

Prompt Engineering: Three distinct strategies were tested: (i) Simple (basic task description), (ii) Moderate (added constraints), and (iii) Advanced (incorporating chain-of-thought reasoning). All prompts required JSON outputs containing a danger level, distance estimate, and a bounding box for spatial grounding.
Fine-Tuning: The models underwent two rounds of parameter-efficient fine-tuning using QLoRA. Stage 1 adapted only attention projections ( $q\_proj, v\_proj$ ), while Stage 2 extended adaptation to all main projections. A training set of 200 balanced images was used, distinct from the test set to prevent data leakage.

Performance was measured using Accuracy, Weighted F1-score, High-Danger Recall (a critical safety metric), and Intersection over Union (IoU) to evaluate spatial grounding.

Key Results

Overall Performance: Without fine-tuning, all models performed near the stratified random baseline (Weighted F1 $\approx$ 0.25). While fine-tuning yielded modest overall improvements for many configurations (Weighted F1 scores in the 0.21–0.24 range), the best-performing configurations achieved higher scores. Specifically, the Qwen model with a simple prompt and a second round of fine-tuning achieved a Weighted F1 of 0.234, while the configuration optimized for high-danger recall (Qwen with an advanced prompt and one round of fine-tuning) achieved a Weighted F1 of 0.211.
High-Danger Detection: A significant disparity emerged between models. Qwen-VL demonstrated substantially higher recall for high-danger cases compared to InternVL and SmolVLM, which remained near zero recall for this class regardless of configuration. Notably, the Qwen configuration using the Advanced prompt and one round of fine-tuning achieved a high-danger recall of 0.432.
Prompt Strategy Trade-offs: The "Advanced" prompt with chain-of-thought reasoning maximized high-danger recall (0.432) but resulted in lower precision for other classes, yielding a Weighted F1 of 0.211. Conversely, the "Simple" prompt with a second fine-tuning round provided the best balanced classification across all levels with a Weighted F1 of 0.234. It is important to note that while the Advanced prompt configuration achieved a high F1 score of 0.296, this metric applies specifically to the 'high' danger class alone, not the overall Weighted F1.
Spatial Grounding: A critical finding is the decoupling of classification accuracy and spatial grounding. While Qwen-VL was the only model to provide useful localization (Mean IoU $\approx$ 0.46 before fine-tuning), correct danger classification did not correlate with better bounding box accuracy. Even for correctly identified high-danger scenes, the model's IoU was not significantly higher than for missed detections.
Impact of Fine-Tuning: Fine-tuning did not improve spatial grounding for any model; in some cases, it slightly degraded performance. For smaller models like SmolVLM, additional fine-tuning appeared detrimental, likely due to catastrophic forgetting.

Significance and Conclusions
The paper concludes that current VLMs are only partially and unevenly capable of assessing proxemic risk from egocentric data. While Qwen-VL shows promise in detecting high-danger situations, particularly when guided by advanced prompting, the other models tested failed to reliably identify these critical scenarios.

The authors emphasize that the ability to produce a correct safety label does not imply the model is attending to the relevant spatial region of the scene. Consequently, four-class proxemic reasoning based on single-view egocentric frames remains an unsolved problem for modern VLMs. The study suggests that while targeted prompting and fine-tuning can improve high-danger detection in specific models, the lack of reliable spatial grounding and the gap between classification and localization limit the immediate deployability of these systems as standalone safety triggers for embodied robots. Future work is directed toward larger datasets and comparisons with more powerful, API-scale models to determine if these limitations are architectural or scale-dependent.

The Experiment: Can AI Spot Danger?

The Results: One Hero, Two Strugglers

What This Means for Robot Safety

Technical Summary: Can Vision-Language Models Assess Proxemic Risk from Egocentric Robot Images?

More like this