🤖 AI

Simulating Human Audiovisual Search Behavior

This paper introduces Sensonaut, a resource-rational computational model of embodied audiovisual search that simulates how humans adaptively balance time, effort, and accuracy under uncertainty, and validates its ability to reproduce human search behaviors and errors against new experimental data.

Original authors: Hyunsung Cho, Xuejing Luo, Byungjoo Lee, David Lindlbauer, Antti Oulasvirta

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hyunsung Cho, Xuejing Luo, Byungjoo Lee, David Lindlbauer, Antti Oulasvirta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are in a crowded, noisy parking lot at night. You need to find your car, which is beeping, but there are dozens of other cars, some with similar beeps, and some parked in front of others blocking your view. You don't just stand still and wait for the perfect moment; you actively turn your head, walk a few steps, and maybe even circle around to get a better angle. You are constantly weighing the effort of moving against the benefit of getting closer to the answer.

This paper introduces Sensonaut, a computer program designed to simulate exactly how humans solve this kind of puzzle. It's not just a map; it's a "digital brain" that learns to search like a person does.

Here is a breakdown of how it works, using simple analogies:

1. The Core Idea: The "Smart Shopper"

Think of Sensonaut as a very smart, but slightly lazy, shopper.

The Goal: Find a specific item (the target) in a messy store (the environment).
The Problem: The shopper can't see everything at once (some items are hidden), and the store is noisy (auditory distractions).
The Strategy: The shopper has a rule: "Don't move unless it's worth the energy."
- If they hear a sound and see a shape that looks right, they might just grab it immediately.
- If they are confused, they will turn their head (a cheap, low-energy move) to get a better listen.
- If turning their head isn't enough, they will walk over (a more expensive, high-energy move) to get a closer look.

The paper calls this "Resource-Rationality." It means the model doesn't try to be perfect; it tries to be efficient. It balances the cost of moving (time and energy) against the reward of finding the target.

2. How the "Digital Brain" Thinks

The model uses a mathematical framework called a POMDP (Partially Observable Markov Decision Process). Let's translate that into everyday terms:

The Fog of War: The model never knows exactly where the target is. It only has a "belief map"—a mental guess of where the target might be.
Two Senses:
- Ears (Audio): Like a radar that works in all directions but is fuzzy on distance. It tells the model, "The sound is coming from roughly over there," but it can't tell if the object is 2 meters away or 10 meters away.
- Eyes (Visual): Like a high-definition camera that only works when you look directly at something. It gives a sharp, clear picture, but if a car is behind another car, the camera sees nothing.
The Update Loop: Every time the model turns its head or takes a step, it gets new data. It mixes this new data with its old guess to update its "belief map." If the new data matches the guess, the belief gets stronger (the fog lifts). If the data is confusing, the belief stays fuzzy.

3. Learning by Trial and Error

The model didn't start out knowing how to search. The researchers used a technique called Reinforcement Learning.

Imagine teaching a dog. If the dog finds the ball, it gets a treat (positive reward). If it walks into a wall or takes too long, it gets a mild shock or no treat (penalty/cost).
Sensonaut played thousands of virtual parking lot games. It learned that:
- Turning your head is cheap and often solves the problem.
- Walking is expensive, so only do it if you are really stuck.
- If you are 90% sure, just commit and stop searching. Don't waste time trying to get to 100%.

4. Did It Work? (The Human Test)

The researchers tested Sensonaut against real humans in a Virtual Reality (VR) parking lot. They created tricky scenarios:

The "Back" Test: The target was behind the person.
The "Crowd" Test: There were many cars.
The "Imposter" Test: There were cars that looked and sounded exactly like the target (distractors).

The Results:

Matching Behavior: Just like humans, Sensonaut took longer to find targets that were behind it or hidden behind other cars.
Matching Effort: When the task was hard, both humans and the model turned their heads more. When it was easy, they didn't move much.
Matching Mistakes: This is the most interesting part. The model made the same mistakes humans did.
- Example: If a target was hidden behind a wall, but a "fake" target was visible in the same direction, both humans and the model sometimes gave up and picked the fake one, thinking, "That's probably it."
- Example: If two cars were far apart but lined up perfectly, both sometimes picked the wrong one because the distance was hard to judge.

5. Where It Differs

The model isn't a perfect clone.

Walking: Humans sometimes take a tiny, exploratory step just to peek around a corner. Sensonaut is a bit more rigid; it only walks if it's absolutely necessary because it calculates the "cost" of walking very strictly.
Precision: Humans can turn their heads by any amount (e.g., 15 degrees). The model turns in fixed 30-degree chunks, like a robot.

Summary

The paper claims that Sensonaut is a successful simulation of how humans search for things using both sight and sound. It proves that human search behavior isn't random; it's a calculated game of "is this movement worth the energy?" By understanding this, the authors say we can build better interfaces (like car systems or VR games) that anticipate where a user will look and how they will move, making technology feel more natural and less frustrating.

The model is available as open-source software, meaning other researchers can use this "digital shopper" to test their own designs before building them.

Technical Summary: Simulating Human Audiovisual Search Behavior

Problem Statement

Locating targets in space using auditory and visual cues is a fundamental challenge in Human-Computer Interaction (HCI), particularly in ubiquitous and immersive environments (e.g., finding a car in a parking lot or a speaker in a virtual meeting). Current computational models of audiovisual search suffer from two primary limitations:

Isolation of Perception and Action: Existing models often treat sensory localization (e.g., binaural hearing) and cognitive decision-making as separate processes. They fail to capture how humans adaptively coordinate movement (head rotation, locomotion) with sensory strategies to resolve perceptual ambiguities like noise, occlusion, and front-back confusion.
Lack of Resource Constraints: Traditional "ideal observer" models (e.g., Bayesian inference) assume optimal, exhaustive search strategies that ignore the physical costs (time, effort, risk of collision) humans face. They do not account for the trade-offs people make between accuracy and the cost of gathering information.

The paper argues that a computational model is needed that unifies sensory inference with resource-rational decision-making under partial observability to predict user behavior, search time, effort, and error modes.

Methodology

The Sensonaut Model

The authors introduce Sensonaut, a computational model of embodied audiovisual search. The core assumption is that people deploy their bodies and sensory systems to efficiently improve their chances of locating a target, trading off time and effort under perceptual constraints.

1. Formal Framework (POMDP):
The task is formalized as a Partially Observable Markov Decision Process (POMDP):

State ( $S$ ): Hidden target location (polar coordinates), agent position/heading, and scene layout (distractors/occluders).
Actions ( $A$ ): Discrete embodied actions: turn_left, turn_right, move_forward, stay, and commit.
Observations ( $O$ ): The agent does not see raw sensory data directly but operates on a posterior belief distribution ( $b_t$ ) over potential target locations. This belief is updated step-by-step based on noisy auditory and visual cues.
Reward ( $R$ ): Defined as Utility ( $U$ ) minus Cost ( $C$ ).
- Utility: Positive reward for correctly identifying the target within tolerance; penalty for errors.
- Cost: Penalties for physical effort (head turns, locomotion), time (waiting), and collisions.
Objective: Learn a policy ( $\pi$ ) that maximizes expected net utility: $\pi^* = \arg\max_\pi E[\sum \gamma^t (U(b_t, a_t) - C(a_t))]$ .

2. Multisensory Perception and Belief Update:
The model maintains a belief distribution over an egocentric polar grid ( $R \times \Theta$ ).

Auditory Likelihood: Based on Interaural Time Differences (ITD). Modeled as a Gaussian distribution over azimuth angles, independent of range. It provides omnidirectional evidence even when the target is occluded.
Visual Likelihood: Provides high-precision, view-dependent evidence. It accounts for occlusion (only the nearest object in a line of sight is visible) and feature similarity. Visual evidence is accumulated over time using an exponential moving average to handle head rotation.
Belief Update: The posterior belief is updated by combining the prior with the joint likelihood of auditory and visual cues in log-space. A "leaky" update mechanism allows the model to maintain multimodal hypotheses (considering distractors) until evidence becomes decisive.

3. Policy Learning:
Since solving the POMDP analytically is intractable, the authors use Reinforcement Learning (RL) with Proximal Policy Optimization (PPO). The agent learns a resource-rational policy through trial and error, balancing the value of information gain against the physical costs of actions.

Data Collection

To validate the model, the authors conducted a controlled Virtual Reality (VR) study with 12 participants.

Task: Participants searched for a specific colored car emitting a beeping sound in a virtual parking lot.
Variables:
- Angle: Initial target direction (Front, Side, Back).
- Complexity: Number of objects (5, 7, 12 cars).
- Distractors: Number of visual distractors (0, 2, 4 cars of the target color).
Metrics: Search time, cumulative head rotation, displacement (walking distance), and localization accuracy.
Protocol: Participants physically walked and turned in a large room while wearing a Meta Quest 3. They committed to a location by pressing a button, after which they estimated the target's position.

Key Results

1. Behavioral Alignment

Sensonaut successfully reproduced key systematic patterns of human search behavior across 270 test maps:

Search Time: Both humans and the model took longer to find targets located behind them or when more distractors were present.
Head Turns: Both increased head rotation significantly when targets were behind or when distractors were present. The model captured the correlation ( $R^2 = 0.58$ ) between human and model head turns, despite the model using fixed 30° steps while humans rotate continuously.
Accuracy: Both maintained high accuracy (~95%), with slight drops in difficult conditions (side/back angles, high distractor counts). The model explained 35% of the variance in human accuracy.
Displacement: The model generally minimized walking more aggressively than humans ( $R^2 = 0.06$ ), reflecting its explicit cost function. Humans showed higher variability in walking distances.

2. Emergent Strategies

The model exhibited three characteristic human-like strategies driven by resource rationality:

Early Commit: When initial evidence was strong, both humans and the model committed immediately without movement.
Head Turns for Utility: When initial evidence was ambiguous (e.g., front-back confusion), both used low-cost head turns to gather information and sharpen the belief distribution before committing.
Locomotion for Disambiguation: When head turns were insufficient (e.g., occluded targets or aligned distractors), both moved to new positions to resolve ambiguity via motion parallax or occlusion changes.

3. Error Reproduction

The model successfully reproduced specific human error modes:

Occlusion-driven errors: Both humans and the model frequently selected a visible distractor when the true target was occluded, leading to false certainty.
Proximity/Depth Confusion: Both were misled by distractors that were closer or farther but aligned in visual angle, selecting the wrong object due to unresolved posterior peaks.
Note: The model did not reproduce "front-back confusions" where humans committed early to a front object while the target was behind; the model consistently performed head turns to resolve this ambiguity.

Significance and Claims

The paper claims that Sensonaut provides a resource-rational account of embodied audiovisual search. Its significance lies in:

Unifying Perception and Action: It bridges the gap between low-level perceptual models and high-level cognitive architectures by integrating cue integration with embodied action costs within a single POMDP framework.
Predictive Capability: It predicts not just success rates, but also the time, effort, and specific error modes associated with search tasks under varying complexities.
Design Utility: The model serves as a simulation tool for designing audiovisual interfaces (e.g., in XR, assistive systems) that minimize search costs and cognitive load. It allows designers to test how spatial layouts, distractors, and guidance cues affect user behavior before deployment.
Validation of Resource Rationality: The results support the hypothesis that human search behavior is not merely a reaction to sensory input but a strategic optimization of information gain against physical and temporal costs.

The authors conclude that while the model does not perfectly match human variability in locomotion (due to coarse action steps and binary visibility assumptions), it captures the essential strategic dynamics of embodied search, offering a principled foundation for simulation-based design in HCI.