🤖 AI

Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI

This survey provides a comprehensive overview of Physical AI by bridging the gap between separate trajectories of physical perception and symbolic reasoning, systematically examining physics-grounded methods across embodied systems and generative models to advocate for next-generation world models that achieve genuine understanding of physical laws for safer and more interpretable AI.

Original authors: Kun Xiang, Terry Jingchen Zhang, Yinya Huang, Jixi He, Zirong Liu, Yueling Tang, Ruizhe Zhou, Lijing Luo, Youpeng Wen, Xiuwei Chen, Bingqian Lin, Jianhua Han, Hang Xu, Hanhui Li, Bin Dong, Xiaodan Lia

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kun Xiang, Terry Jingchen Zhang, Yinya Huang, Jixi He, Zirong Liu, Yueling Tang, Ruizhe Zhou, Lijing Luo, Youpeng Wen, Xiuwei Chen, Bingqian Lin, Jianhua Han, Hang Xu, Hanhui Li, Bin Dong, Xiaodan Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the real world, not just as a collection of pictures, but as a place where things have weight, bounce, crash, and follow rules like gravity. This paper, "Aligning Perception, Reasoning, Modeling, and Interaction: A Survey on Physical AI," argues that current AI is like a student who has memorized the answers to a math test but doesn't actually understand why the math works.

The authors propose a four-step "training curriculum" to turn AI from a passive observer into a master of physical reality. They call this Physical AI.

Here is the journey, explained through simple analogies:

1. Physical Perception: The "Eyes and Hands"

The Problem: Current AI can look at a photo of a ball and say, "That's a red ball." But if you drop the ball, it doesn't know it will bounce. It sees the image, not the physics.
The Analogy: Think of this as a child learning to touch. First, they learn what a toy looks like (Object Recognition). Then, they learn where it is in the room (Spatial Perception). Next, they learn that a rubber ball feels bouncy while a rock feels hard (Intrinsic Properties). Finally, they learn that if you push the ball, it rolls (Dynamic Estimation).
The Goal: To move from just "seeing" a picture to understanding the hidden rules of the object, like its weight, texture, and how it moves.

2. Physics Reasoning: The "Brain"

The Problem: Once the AI sees the ball, it needs to understand why it moves. Current AI often guesses based on patterns (e.g., "I've seen balls roll before, so this one will too"). It doesn't truly understand the cause.
The Analogy: This is like moving from a toddler who drops a spoon to a physicist who can explain why the spoon fell.

Symbolic Reasoning: Solving a math problem on paper (e.g., "If a car goes 25 m/s...").
Multimodal Reasoning: Looking at a diagram of a bridge and explaining why it might collapse.
Causal Reasoning: Asking, "What would happen if I didn't brake?" (Counterfactuals).
The Goal: To stop guessing and start using the "laws of the universe" (like gravity or friction) to explain what is happening.

3. World Modeling: The "Imagination"

The Problem: If you ask an AI to predict the future, it might hallucinate (make things up). It might draw a car floating in the air because it looks cool, not because it's physically possible.
The Analogy: This is the AI's "dreaming" ability. A good world model is like a movie director who can simulate a scene in their head before filming it. They can ask, "If I drop this vase, will it shatter?" and run a mental simulation to see the answer.
The Goal: To build a mental "sandbox" where the AI can predict the future or reconstruct the past with perfect physical accuracy, ensuring that if a car hits a wall, it crumbles realistically, not magically.

4. Embodied Interaction: The "Body"

The Problem: You can have a smart brain and a great imagination, but if the robot's arm is clumsy or it doesn't know how to grip a slippery cup, it fails.
The Analogy: This is the moment the robot actually does something. It's like the difference between a chess grandmaster who can only talk about moves and one who can actually play the game against a human.

Robotics: Picking up objects without crushing them.
Navigation: Walking through a crowded room without bumping into people.
Autonomous Driving: Driving a car that reacts to a sudden rainstorm or a child running into the street.
The Goal: To close the loop. The robot perceives the world, reasons about it, simulates the outcome, and then acts on it safely.

The Big Picture: Why This Matters

The paper argues that we can't just jump to step 4 (making robots drive cars). We need to build these skills in order, like climbing a ladder:

Perception gives us the raw data.
Reasoning turns that data into understanding.
Modeling lets us predict what happens next.
Interaction lets us change the world based on those predictions.

The Current Reality Check:
The authors admit that right now, most AI is stuck at the bottom of the ladder. It's great at recognizing patterns (like spotting a cat in a photo) but terrible at understanding the physics (like knowing a cat can't walk through a wall).

They conclude that to build truly safe and reliable AI for the real world, we need to stop just feeding it more data and start teaching it the "rules of the game" (the laws of physics) so it can truly understand, predict, and interact with the world around it.

In short: The paper is a roadmap for teaching AI to stop being a "photographer" that just takes pictures of the world and start being a "physicist" that understands how the world actually works.

1. Problem Statement

Current Artificial Intelligence systems, despite achieving superhuman performance in pattern recognition (e.g., image classification), fundamentally struggle with physical understanding. They lack the ability to grasp causal relationships, predict dynamic outcomes, or interact robustly with the physical world.

The Gap: While humans intuitively master physical laws from early childhood, frontier AI models often fail at basic physical reasoning tasks (e.g., predicting a bouncing ball's trajectory or understanding collision dynamics).
The Limitation: Existing surveys treat physical capabilities (perception, reasoning, modeling, interaction) as isolated research silos. There is a lack of a unified framework that explains how these capabilities evolve from passive observation to active physical comprehension.
The Core Challenge: Moving AI from statistical correlation (learning from massive data) to causal knowledge rooted in physical principles to enable safe, generalizable, and interpretable deployment in robotics, autonomous driving, and scientific discovery.

2. Methodology and Framework

The authors propose a capability–task hierarchy that organizes Physical AI into four interconnected, progressive stages. This framework moves beyond architectural specifics (e.g., Transformers vs. GNNs) to focus on the principal output of the system.

The Four Stages of Physical AI:

Physical Perception:
- Goal: Transform raw sensory data into structured physical representations.
- Tasks: Object recognition, spatial perception (2D/3D relations), intrinsic property estimation (mass, material, rigidity), and dynamic estimation (motion, contact forces).
- Key Insight: This is the grounding layer; without accurate physical state estimation, higher-level reasoning fails.
Physics Reasoning:
- Goal: Derive explanations and solutions using symbolic laws and causal mechanisms.
- Tasks: Symbolic reasoning (textbook/exam problems), multimodal-grounded reasoning (diagrams/videos), causal/counterfactual reasoning ("what if" scenarios), and accelerating physics research (hypothesis generation).
- Key Insight: Moves from "what is happening" to "why it happens," utilizing Chain-of-Thought (CoT) and tool-augmented agents.
World Modeling:
- Goal: Construct a physically coherent, generative model of the environment to predict future states.
- Tasks: Image/video generation (physically plausible), scene reconstruction (3D/4D), and physics-constrained simulation.
- Key Insight: Unlike pure perception, world modeling builds a reusable, simulatable representation of the environment, bridging the gap between observation and prediction.
Embodied Interaction:
- Goal: Execute actions that intervene in the physical environment based on the previous three stages.
- Tasks: Robotics (manipulation), Navigation (object-goal, VLN), and Autonomous Driving.
- Key Insight: Closes the loop by deploying policies in real-world or high-fidelity simulated environments, requiring real-time adaptation to physical constraints.

Taxonomy and Analysis:

The survey analyzes over 300 papers spanning 2012–2026.
It employs a PRISMA-style systematic review process to categorize methods, benchmarks, and datasets.
The authors distinguish between physics-inspired architectures (e.g., Boltzmann machines) and physics-aware AI systems (the focus of this survey), which use deep learning to solve physics problems end-to-end.

3. Key Contributions

Unified Framework: The paper introduces the first survey to systematically connect perception, reasoning, modeling, and interaction as a progressive developmental pathway, rather than isolated capabilities.
Comprehensive Taxonomy: A detailed three-tier hierarchy (Capability $\to$ $\to$ Task Family $\to$ $\to$ Subtask) covering:
- Perception: From object detection to dynamic interaction inference.
- Reasoning: From symbolic derivation to causal discovery.
- Modeling: From static rendering to counterfactual simulation.
- Interaction: From robotic control to autonomous driving planning.
Critical Gap Analysis: The authors provide evidence that current systems often fail to generalize across stages. For instance, a model strong in video generation (World Modeling) may still fail at causal reasoning (Reasoning) or real-world control (Interaction).
Resource Repository: The authors maintain a continuously updated taxonomy and benchmark repository (Awesome-AI-for-Physics on GitHub) to facilitate future research.

4. Results and Findings

Performance Disparities:
- Perception: Modern MLLMs (e.g., GPT-4V) excel at semantic recognition but struggle with precise geometric grounding and physical anomaly detection (e.g., Phys-AD benchmarks show ~52% AUROC for anomalies).
- Reasoning: While LLMs can solve textbook problems, they fail on complex, multi-step physical reasoning requiring visual grounding (e.g., SeePhys benchmark shows a significant drop in performance when vision is essential).
- Modeling: Video generation models (e.g., Sora, Kling) produce visually coherent sequences but often violate physical laws (e.g., momentum conservation, object permanence) when evaluated on physics-specific benchmarks like PhysicsIQ.
- Interaction: Embodied agents often rely on simulation-to-reality transfer, which remains fragile due to the "Sim-to-Real" gap where learned policies fail in noisy, real-world conditions.
Evidence for Progression:
- Positive: Systems that integrate earlier stages (e.g., using world models to inform planning in autonomous driving) show improved robustness.
- Negative: Mastery of one stage does not guarantee success in the next. High-performing perception models do not automatically possess causal reasoning capabilities.
Sim-to-Real Gap: The survey highlights that current world models optimize for perceptual plausibility rather than physical consistency, leading to failures when deployed in real-world embodied tasks.

5. Significance and Future Directions

Paradigm Shift: The paper argues for a shift from isolated, task-specific solutions to integrated architectures that traverse the full spectrum from perception to interaction.
Architectural Recommendations:
- Neuro-Symbolic Integration: Combining neural perception with explicit symbolic reasoning to enforce physical laws.
- Differentiable Physics: Using differentiable engines to allow end-to-end optimization of physical parameters.
- Embodied Learning: Moving beyond passive observation to active intervention, where agents learn physical laws through interaction and feedback.
Impact: This framework provides the foundation for next-generation Physical AI capable of:
- Safe Deployment: Reducing risks in robotics and autonomous driving by ensuring adherence to physical constraints.
- Generalization: Enabling AI to handle novel scenarios and counterfactuals (e.g., "what if the road was icy?").
- Scientific Discovery: Automating hypothesis generation and experimental design in physics and materials science.

In conclusion, the survey posits that true Physical AI requires the tight integration of all four capabilities, where perception grounds state, reasoning abstracts laws, modeling projects trajectories, and interaction closes the loop through experience.