🤖 AI

Grounding Social Perception in Intuitive Physics

This paper proposes that human social perception is grounded in the integration of intuitive psychology and physics, demonstrating through the PHASE dataset and the SIMPLE computational model that accurately inferring agents' goals and relationships from physical interactions requires a physics-grounded Bayesian inverse planning approach rather than simple visual pattern matching.

Original authors: Lance Ying, Aydan Y. Huang, Aviv Netanyahu, Andrei Barbu, Boris Katz, Joshua B. Tenenbaum, Tianmin Shu

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lance Ying, Aydan Y. Huang, Aviv Netanyahu, Andrei Barbu, Boris Katz, Joshua B. Tenenbaum, Tianmin Shu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a cartoon where a red triangle and a blue circle are moving around a room. Even though they are just shapes, you instantly "know" what's happening: "Oh, the red triangle is chasing the blue circle!" or "They are working together to push a box."

You don't need to see faces or hear voices to understand this. You understand it because you have a deep, intuitive understanding of how the physical world works (physics) and how people think (psychology).

This paper, titled "Grounding Social Perception in Intuitive Physics," argues that our brains don't just watch videos like a camera; they run a mental simulation in the background to figure out what's going on.

Here is the breakdown of their discovery, using simple analogies:

1. The Problem: The "Magic" of Social Understanding

When we watch the classic "Heider-Simmel" movie (where shapes move around), we instantly invent stories. We see a "chase," a "help," or a "fight."

Previous computer models tried to do this in two ways, and both failed:

The "Pattern Matcher" (AI Vision): These models are like a student who memorizes thousands of photos. They see "two shapes moving close together" and guess "friendship." But if you change the physics slightly (e.g., make the wall slippery), they get confused. They don't understand why the shapes are moving; they just recognize the pattern.
The "Symbolic Planner" (Old Logic AI): These models are like a chess player who knows the rules but has no sense of gravity. They know "Agent A wants to go to Point B," but they don't understand that "Agent A is too heavy to jump over that wall." They miss the physical reality.

2. The Solution: The "Mental Movie Director"

The researchers propose that humans are like Mental Movie Directors. When we see an action, we don't just look at it; we run a simulation in our heads to ask: "If I were that agent, with those physical limits, trying to achieve that goal, what would I do?"

If the simulation matches what we see, we understand the scene. If it doesn't, we try a different story.

3. The New Dataset: "PHASE" (The Training Gym)

To test this, the team created a massive library of 500 short animations called PHASE.

The Setup: Imagine a 2D video game with two trapezoid-shaped agents, some round objects, and four corner landmarks.
The Rules: The agents have different "strengths" (some can push heavy objects, some can't). They have limited vision. They can be friends, enemies, or neutral.
The Goal: The researchers generated thousands of scenarios where agents chase, block, help, or compete with each other, all governed by strict laws of physics (collisions, momentum, obstacles).

They then asked real humans to watch these videos and label the interactions. The humans were incredibly consistent, proving that even with simple shapes, we all agree on the "story" because we all share the same intuitive physics.

4. The New Model: "SIMPLE" (The Smart Detective)

The team built a computer model called SIMPLE (Simulation, Planning, and Local Estimation) to mimic the human "Mental Movie Director."

Instead of just looking at the video, SIMPLE does this:

Hypothesize: It guesses, "Maybe the red agent is trying to push the blue ball to the yellow corner, and the green agent is trying to stop it."
Simulate: It runs a physics engine in its "mind." It asks, "If the red agent is that strong and the green agent is that fast, would their movements look like the video I'm watching?"
Check: It compares its mental simulation to the actual video.
- Does the simulation match? Great! That's the answer.
- Does it look weird? (e.g., The simulation shows the red agent getting stuck, but the video shows them moving freely). Then, SIMPLE changes its guess: "Okay, maybe the red agent is actually stronger than I thought," or "Maybe they are working together."
Repeat: It does this thousands of times until it finds the story that best explains the physics of the scene.

5. The Results: Why It Matters

When they tested SIMPLE against other AI models:

The "Pattern Matchers" (like SocialGNN and Gemini 2.5 Pro): They struggled. They often thought two agents fighting over an object were "helping" each other because they were moving in the same direction. They lacked the physical common sense to realize they were pulling in opposite directions.
The "Symbolic Planners": They failed to understand why an agent couldn't move a heavy object, leading to wrong guesses about their goals.
SIMPLE: It performed almost as well as humans. It correctly identified that two agents pulling a ball in opposite directions were "competing," not "collaborating," because it understood the forces involved.

The Big Takeaway

This paper proves that to truly understand social behavior (even in simple cartoons), you cannot separate physics from psychology.

Psychology tells us what an agent wants (the goal).
Physics tells us what is possible (the constraints).

You can't know if someone is "helping" or "hindering" unless you understand the physical forces they are applying. Our brains naturally combine these two worlds, and the SIMPLE model is the first AI to successfully do the same by running a "mental physics simulation" to solve social puzzles.

In short: To understand people, you have to understand the world they are moving through. You can't just watch the dance; you have to feel the music and the floor.

1. Problem Statement

Human social perception is deeply rooted in an understanding of the physical world. When observing agents (people or objects), humans do not merely match visual patterns; they infer rich social information (goals, relationships, intentions) by reasoning about intuitive physics (forces, obstacles, motion constraints) and intuitive psychology (rationality, purpose).

Existing computational models for social perception fall into two categories, both of which have limitations:

Symbolic Inverse Planning: These models assume agents are rational planners but often abstract away physical dynamics, treating actions as symbolic state transitions. They fail to account for how physical constraints (e.g., an agent's strength or object mass) influence behavior.
End-to-End Neural Models: These models (e.g., Graph Neural Networks, Vision-Language Models) learn from large datasets of visual scenes. While powerful, they rely on feedforward pattern matching, lack explicit physical reasoning, struggle to generalize to unseen physical configurations, and fail to capture the "analysis-by-synthesis" process humans use to infer mental states.

The core question addressed is: How can we computationally model human social perception that is grounded in the integration of intuitive physics and intuitive psychology?

2. Methodology

The authors propose a two-pronged approach: a new dataset for training/evaluation and a novel computational model.

A. The PHASE Dataset

PHASE (PHysically grounded Abstract Social Events) is a large-scale dataset of procedurally generated animations designed to test social perception in physically grounded environments.

Style: Inspired by the classic Heider and Simmel movie, featuring 2D geometric shapes (trapezoids as agents, circles as objects, squares as landmarks).
Physics: The environment uses a physics engine where agents have varying sizes, strengths (force capacity), and limited fields of view. Objects have mass and can be moved, blocked, or carried.
Scenarios: The dataset includes 500 videos covering four scenario types based on agent goals and relationships:
1. Collaborative: Agents work together (Friendly).
2. Independent: Agents pursue non-conflicting goals (Neutral).
3. Competing: Agents fight over the same resource (Adversarial).
4. Opposing: One agent actively hinders the other (Adversarial).
Annotation: Human participants consistently labeled these videos with 23 distinct interaction types (e.g., chasing, blocking, helping), establishing a ground truth for evaluation.

B. The SIMPLE Model

The authors introduce SIMPLE (SIMulation, Planning, and Local Estimation), a physics-grounded Bayesian inverse planning model.

Core Hypothesis: Human observers invert a joint physical and social generative model. They simulate possible scenarios (analysis-by-synthesis) to find the latent variables that best explain the observed trajectories.
Generative Model ( $G$ ):
- Hierarchical Planner (Intuitive Psychology): Samples high-level goals and subgoals based on agent beliefs and social relationships.
- Forward Physics Engine (Intuitive Physics): Simulates the physical consequences of actions, resolving contact dynamics, collisions, and object manipulation.
Inference Process:
- Goal: Infer the posterior probability of latent variables $h = \langle g_i, g_j, \alpha_{ij}, \alpha_{ji}, f_i, f_j \rangle$ , where $g$ are goals, $\alpha$ are social relations, and $f$ are physical strengths.
- Algorithm: Uses Metropolis-Hastings MCMC with a "bottom-up" proposal mechanism.
  - It proposes hypotheses based on observed trajectory segments.
  - It simulates the trajectory $\hat{s}_{1:T}$ for each hypothesis.
  - It calculates the likelihood $P(s_{1:T}|h)$ based on the deviation between the simulated and observed trajectories.
  - Local Estimation: To handle high-dimensional continuous spaces, the model focuses on local time intervals where simulation errors are highest to update hypotheses, rather than relying solely on global error.
Output: Marginalized probability distributions for specific goals and social relations.

3. Key Contributions

PHASE Dataset: A controlled, procedurally generated dataset that systematically varies physical parameters (geometry, dynamics, agent capacities) and social structures, bridging the gap between abstract animations and real-world physics.
SIMPLE Model: A computational framework that unifies Bayesian inverse planning with physical simulation. It demonstrates that inferring social states requires inverting a model that explicitly accounts for physical constraints.
Empirical Validation: A comprehensive comparison showing that integrating physics into social reasoning is necessary for human-level performance, outperforming both symbolic planners and modern neural baselines.

4. Results

The authors evaluated SIMPLE against human participants, end-to-end neural baselines (SocialGNN, Gemini 2.5 Pro), and a physics-agnostic predicate inverse planning model on 100 test videos.

Accuracy:
- SIMPLE achieved the highest accuracy in both goal classification (0.78) and relation classification (0.91), matching human performance levels (Human: 0.78 goal, 0.89 relation).
- Baselines: Neural models (SocialGNN, Gemini 2.5 Pro) and the predicate-only model performed significantly worse, particularly in complex competing and opposing scenarios where physical dynamics (e.g., "tug-of-war" forces) are critical.
Human Alignment:
- SIMPLE showed a strong correlation with human judgment distributions ( $r=0.82$ for goals, $r=0.78$ for relations), whereas baselines had correlations below 0.6.
- SIMPLE successfully captured human uncertainty patterns (e.g., higher uncertainty in goal inference than relation inference).
Qualitative Analysis:
- In "tug-of-war" scenarios, neural models often misclassified competitive interactions as collaborative because the agents moved together. SIMPLE correctly identified the conflicting forces and opposing goals by simulating the physics of the interaction.
- The predicate model succeeded in simple "independent" scenarios but failed when fine-grained physical constraints (e.g., "Can this agent push that object?") were required.

5. Significance and Discussion

Mechanism of Social Cognition: The paper provides strong evidence that human social perception is not just visual pattern matching but a generative inference process that simulates physical laws. To understand why an agent acts a certain way, one must understand what is physically possible for them.
Beyond Feedforward Models: The results challenge the efficacy of purely feedforward deep learning models for complex social reasoning. These models require massive data and fail to generalize to novel physical constraints, whereas SIMPLE generalizes via its internal physics engine without task-specific training.
Integration of Core Knowledge: The work supports the theory that human cognition relies on "core knowledge" systems (intuitive physics and psychology) that are tightly coupled. SIMPLE operationalizes this coupling, suggesting that future AI systems for social understanding must incorporate explicit physical reasoning.
Limitations & Future Work:
- Computational Cost: The MCMC simulation approach is computationally expensive compared to real-time human perception. Future work may explore "amortized inference" (training neural nets to approximate the Bayesian posterior).
- 2D Simplification: The current model uses 2D shapes. Extending this to 3D articulated bodies and gaze is a necessary step for real-world robotics and human-robot interaction.

In conclusion, this paper establishes that grounding social perception in intuitive physics is essential for accurate and human-like inference of agent goals and relationships, offering a robust computational framework (SIMPLE) that outperforms current state-of-the-art neural and symbolic approaches.