🤖 AI

SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models

This paper introduces SafetyALFRED, a new benchmark extending the ALFRED environment with real-world kitchen hazards to evaluate multimodal large language models, revealing a significant gap between their ability to recognize safety risks in static question-answering settings and their capacity to actively mitigate those risks through embodied planning.

Original authors: Josue Torres-Fonseca, Naihao Deng, Yinpei Dai, Shane Storks, Yichi Zhang, Rada Mihalcea, Casey Kennington, Joyce Chai

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Josue Torres-Fonseca, Naihao Deng, Yinpei Dai, Shane Storks, Yichi Zhang, Rada Mihalcea, Casey Kennington, Joyce Chai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have hired a very smart, highly educated robot butler named "AI." You've taught it everything about the world: how to cook, how to clean, and even the rules of safety from thousands of books and websites.

You give it a simple job: "Go to the kitchen and wash the butter knife."

The robot looks around, sees a smartphone sitting in the sink, and says, "I see a phone in the sink! That's dangerous because water and electricity don't mix. I should move the phone."

So far, so good. The robot is smart.

But then, you ask the robot to actually do the job. It looks at the phone, nods, and immediately turns on the faucet to wash the knife, completely ignoring the phone. Splash! The phone gets ruined.

This is exactly the problem researchers at the University of Michigan and Boise State University discovered in their new paper, SafetyALFRED.

The Big Idea: "Knowing" vs. "Doing"

The researchers built a test called SafetyALFRED. Think of it like a driving test for robots, but instead of a car, they are testing AI agents in a virtual kitchen. They created six types of "kitchen disasters" waiting to happen, such as:

A phone in the sink (Water + Electronics = Bad).
A metal spoon in the microwave (Metal + Microwaves = Fire/Explosion).
A cabinet door left open (Trip hazard).
A stove left on (Fire hazard).

They tested 11 of the smartest AI models available (like Qwen, Gemma, and Gemini) to see if they could handle these situations.

The Two Tests

The researchers ran two different tests to see how the robots behaved:

1. The "Quiz Show" Test (Question & Answer)
They showed the robot a picture of the kitchen and asked, "Is there anything dangerous here?"

Result: The robots were amazing! They got about 90% of the answers right. They could spot the phone in the sink or the open stove almost perfectly. They were like students who aced the safety exam.

2. The "Real Life" Test (Embodied Planning)
They gave the robot the same kitchen scene but told it, "Now, go wash the knife." The robot had to plan its steps and actually move things around.

Result: The robots failed miserably. Even though they knew the phone was dangerous, they often forgot to move it and just tried to wash the knife. Their success rate dropped to less than 60%, and for some tricky hazards, it was near 0%.

The Analogy: The "Know-It-All" Student

Imagine a student who studies hard for a fire safety exam.

In the classroom (The Quiz): The teacher asks, "What do you do if you see a fire?" The student shouts, "Call 911 and use an extinguisher!" They get an A+.
In the real world (The Embodied Task): A trash can actually catches fire. The student, panicked and focused on finishing their homework, accidentally kicks the trash can, making the fire worse, because they were so focused on the "homework" (the task) that they forgot the "safety rule" (the hazard).

The paper calls this the "Alignment Gap." The AI's knowledge of safety doesn't match its actions when it's busy trying to get a job done.

Why Did They Fail?

The researchers found a few reasons for this failure:

Task Tunnel Vision: When the robot is told to "wash the knife," it gets so obsessed with that goal that it ignores the phone in the sink. It's like a driver so focused on getting to the store that they run a red light.
Perception Issues: Without extra help (like a text description of what objects are), the robots sometimes couldn't "see" the danger clearly enough in the image to act on it.
Prioritization: The robots seem to think, "I must finish the task first, safety can wait."

The Solution: The "Safety Co-Pilot"

To fix this, the researchers tried a clever trick. Instead of asking one robot to do everything, they used two robots working together:

The Safety Judge: A robot whose only job is to look at the scene and shout, "Hey! There's a phone in the sink!"
The Doer: The robot that actually washes the knife.

When the "Safety Judge" told the "Doer" about the danger, the "Doer" suddenly got much better at fixing it. It's like having a co-pilot in a car who yells, "Brake!" while you are focused on the GPS.

The Takeaway

The main message of this paper is simple but scary: Just because an AI can pass a safety test doesn't mean it will be safe in the real world.

We can't just ask robots, "Do you know what's safe?" and assume they will act safely. We need to build systems where safety is a constant, active part of their decision-making, not just a fact they memorized for a quiz.

The researchers have released their code and data (SafetyALFRED) so other scientists can build better, safer robots that won't accidentally ruin your phone or start a fire while trying to make you dinner.

1. Problem Definition

The paper addresses a critical gap in the deployment of Multimodal Large Language Models (MLLMs) as autonomous embodied agents. While MLLMs excel at high-level reasoning and task planning, their ability to proactively identify and mitigate physical safety hazards in dynamic, interactive environments remains unproven.

Existing safety benchmarks (e.g., ASIMOV, MM-SafetyBench) primarily evaluate hazard recognition through static, disembodied Question-Answering (QA) tasks. These evaluations fail to test whether an agent can translate abstract safety knowledge into corrective physical actions (mitigation) while simultaneously executing a primary task. The core problem is the alignment gap: models may correctly identify a hazard in a static image but fail to prioritize its removal when planning a sequence of actions in a simulated environment.

2. Methodology

2.1 The SafetyALFRED Benchmark

The authors introduce SafetyALFRED, an extension of the ALFRED benchmark (based on the AI2-THOR simulator).

Environment: 30 kitchen environments with 5 task types (move, stack, wash, heat, cool).
Hazard Injection: The benchmark introduces six categories of real-world kitchen hazards by perturbing the environment state:
1. Appliance Misuse: Metallic/flammable objects in a microwave.
2. Spoilage: Fridge/Cabinet doors left open.
3. Fall/Trip Hazard: Cabinet doors left open or objects on the floor.
4. Fire Hazard: Active stove burners near targets.
5. Property Damage: Water-sensitive objects in a sink.
6. Unsanitary: Objects on dirty floors.
Data Generation: Ground-truth trajectories were generated using PDDL (Planning Domain Definition Language) modifications to ensure that successful task completion requires the mitigation of hazards before proceeding.

2.2 Evaluation Framework

The study evaluates 11 state-of-the-art MLLMs (Qwen 2.5/3 VL, Gemma 3, Gemini 1.5/2.5) across two distinct settings to measure the alignment between knowledge and action:

QA Setting (Hazard Recognition):
- The model acts as a static "safety judge."
- Input: Goal, action history, and scene observation (Vision-only or Metadata-augmented).
- Output: Binary decision (Yes/No) and description of the hazard.
- Metric: Hazard Detection Accuracy ( $Acc_{QA}$ ).
Embodied Setting (Hazard Mitigation):
- The model acts as an agent executing the task.
- Input: Same as QA, plus the requirement to generate the next action and subgoal.
- Constraint: The agent must prioritize hazard removal ( $R_{safe}$ ) over task advancement ( $G$ ) if a hazard is present.
- Metric: Mitigation Success Rate (MSR) and Safety Alignment Rate ( $A$ ).

2.3 Observation Modes

To isolate perception from reasoning, the authors tested two input modes:

Vision-Only (V): Raw RGB images.
Metadata-Augmented (D): Textual descriptions of object states (e.g., "stove is on," "phone is in sink") provided alongside images.

2.4 Multi-Agent Framework

To investigate if the failure is due to task interference, the authors proposed a Multi-Agent System where a dedicated "Safety Judge" (QA agent) identifies hazards and passes this information to the "Embodied Agent" (Actor), decoupling recognition from planning.

3. Key Results

3.1 The Alignment Gap

The most significant finding is the significant misalignment between static recognition and dynamic mitigation:

QA Performance: Models achieved high accuracy in recognizing hazards in static QA settings (up to 92% for top models with metadata).
Embodied Performance: The same models struggled to mitigate these hazards during task execution. Average mitigation success rates were <60% even with metadata, and often <20% in vision-only settings.
Conclusion: High QA accuracy is a poor proxy for embodied safety. Knowing a hazard exists does not guarantee the agent will act to remove it.

3.2 Perception vs. Planning Bottlenecks

Perception: Models struggle to detect hazards from raw pixels (Vision-only), particularly for "Appliance Misuse" and "Property Damage." Metadata significantly improved recognition, suggesting perception is a bottleneck.
Planning: Even with perfect metadata (perfect perception), models failed to mitigate hazards. This indicates the failure is planning-centric: models prioritize the primary task goal over safety constraints, failing to synthesize safety knowledge into action.

3.3 Model Scaling and Bias

Scaling: Larger models generally performed better in QA but did not proportionally improve in embodied mitigation.
Conservative Bias: Models exhibited a high false-positive rate in safe environments (hallucinating hazards >50% of the time in QA), yet failed to act on the actual hazards when they occurred in the embodied setting.

3.4 Multi-Agent Improvement

Decoupling the roles via a multi-agent framework improved mitigation rates (e.g., Qwen 3 VL-32B improved from 0.7% to 71.1% for appliance misuse). However, this did not fully resolve the issue, as the embodied agent still failed to act on the judge's correct warnings in many cases, suggesting a deeper issue in action selection logic.

4. Key Contributions

SafetyALFRED Benchmark: The first benchmark specifically designed to evaluate the alignment between static hazard recognition and dynamic, embodied hazard mitigation in household tasks.
Empirical Evidence of Misalignment: Demonstrated that current SOTA MLLMs possess abstract safety knowledge but lack the ability to operationalize it in physical planning, challenging the efficacy of QA-based safety evaluations.
Analysis of Failure Modes: Identified that failures are not solely due to perception errors but stem from task interference (prioritizing goals over safety) and planning deficits.
Open Source: Released the dataset, code, and evaluation pipeline to facilitate future research in embodied AI safety.

5. Significance and Future Directions

The paper argues that the current paradigm of evaluating MLLM safety solely through static QA is insufficient for real-world robotics.

Paradigm Shift: Safety research must move toward embodied benchmarks that test corrective actions in context.
Data Needs: There is a critical need for more embodied safety data to train models to synthesize hazard recognition with task planning.
Evaluation Metrics: Future evaluations must account for the trade-off between task success and safety, rather than treating them as separate metrics.
Deployment: The findings highlight the risks of deploying large models in physical environments where "knowing" a risk is not enough; the agent must be capable of "doing" the right thing under pressure.

In summary, SafetyALFRED reveals that while MLLMs are becoming better at seeing danger, they are not yet reliable at acting safely, necessitating a fundamental shift in how safety is benchmarked and trained for embodied agents.