📄 animal behavior and cognition

Multisensory Continuous Psychophysics: Perceived Visual Object Location is Improved by Auditory Cues

This pre-registered study demonstrates that continuous psychophysics combined with Kalman filter analysis effectively reveals how auditory cues significantly reduce sensory noise and improve the precision of both manual and ocular tracking of visual objects, particularly under conditions of high visual uncertainty.

Original authors: Jörges, B., Kim, J.-J., Harris, L. R.

Published 2026-06-08

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Jörges, B., Kim, J.-J., Harris, L. R.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are playing a video game where you have to keep your eyes and your controller locked onto a moving target, like a drone or a swarm of flies, in a beautiful virtual meadow. Usually, if the screen gets foggy, you lose your grip on where the target is. But what if you could hear the target buzzing nearby?

This study is like a high-tech version of that game, designed to see how our brains mix what we see with what we hear. The researchers used a new way of testing called "Continuous Psychophysics." Instead of asking you to guess "left or right" in a series of quick, separate questions (like a multiple-choice quiz), this method asks you to constantly follow the object, like a cat chasing a laser pointer. This gives a much smoother, more natural picture of how your brain works in real-time.

Here is how they set up the experiment:

The Foggy Meadow: They created four levels of "fog" in the virtual world. It ranged from a crystal-clear day to a thick, impenetrable fog where you couldn't see the target at all.
The Sound Cue: Sometimes, the target made a sound; other times, it was silent.
The Tracking: Two groups of people (30 in each) tried to follow the target using a controller (their hands) while their eye movements were recorded.

What they found:

Think of your brain as a detective trying to solve a mystery. When the fog is thick (low visibility), the detective has very little visual evidence. However, when the sound is present, it's like the detective suddenly gets a crucial clue from a witness.

Sound Helps in the Fog: When the fog was heavy, having the sound made a huge difference. The participants were much better at guessing where the target was, even if they couldn't see it. The sound acted as a safety net, filling in the gaps left by the missing visual clues.
The "Noise" Meter: The researchers used a special mathematical tool (a Kalman filter) to act like a noise meter. They wanted to know how "noisy" or confused the brain's senses were. They found that when sound was added, the brain's internal "noise" dropped significantly. It was as if the sound turned down the static on a radio, making the signal clearer, whether the person was tracking with their hands or their eyes.
Hands and Eyes Work Together (But Differently): Generally, if someone was good at tracking with their hands, they were also good at tracking with their eyes. They seemed to be using the same "muscle" of attention. However, when the researchers looked specifically at the benefit of the sound, this connection broke down. The sound helped the hands and the eyes in unique ways that weren't directly linked to each other.

The Bottom Line:

This study shows that this new "continuous" way of testing is excellent for understanding how we combine senses. It proves that our brains are smart enough to use sound to sharpen our vision, especially when things get blurry. By using math to measure the "noise" in our senses, the researchers confirmed that sound doesn't just distract us; it actually cleans up the signal, helping us locate objects more accurately when our eyes aren't enough.

Technical Summary: Multisensory Continuous Psychophysics

Problem and Motivation
Traditional psychophysical designs often rely on discrete trials, which may constrain the study of dynamic perceptual processes. Continuous Psychophysics, a paradigm coupling continuous stimuli with continuous responses, offers a promising alternative to overcome these limitations. While previous work, such as Tonelli et al. (2025), has established the utility of this approach, there remains a need to rigorously evaluate its applicability to multisensory integration. Specifically, it is unclear how well this paradigm captures the interplay between visual and auditory cues in dynamic tracking tasks, and whether it can effectively disentangle sensory noise from motor execution. This study aims to expand the scope of Continuous Psychophysics by investigating multisensory integration while simultaneously examining the role of eye movements and employing advanced computational modeling.

Methodology
The study utilized a pre-registered design involving two cohorts of participants ( $n = 30$ each) immersed in a virtual reality (VR) "meadow" environment. The core task required participants to continuously track a moving target using a controller while their eye movements were recorded. Two distinct experimental conditions were employed:

Experiment 1: Tracking a single drone.
Experiment 2: Tracking a swarm of flies.

The experimental design manipulated two primary factors:

Visual Reliability: Four levels of fog were applied, ranging from a completely clear view to impenetrable fog where no visual cues regarding the target's position were available.
Auditory Cues: The presence or absence of sound cues emitted from the moving object.

To analyze the behavioral data, the authors employed a Kalman filter to estimate the underlying sensory noise parameters governing the tracking behavior. This allowed for the separation of sensory uncertainty from motor response characteristics.

Key Results
The analysis yielded several critical findings regarding the interaction between sensory modalities and the nature of tracking performance:

Multisensory Enhancement: The maximum correlation between the stimulus and the participant's response was significantly higher when auditory cues were present, particularly under conditions of high visual uncertainty (heavy fog).
Tracking Delay: Unlike correlation, the tracking delay remained unaffected across all levels of visual fog, suggesting that the temporal lag in response was robust to changes in visual reliability.
Sensory Noise Estimation: The Kalman filter analysis provided strong evidence that sensory noise was lower when sound was present compared to when it was absent. This reduction in noise was observed in both manual (controller) and ocular (eye movement) tracking, with the effect being most pronounced in conditions of high visual uncertainty.
Correlation Between Modalities: Exploratory analyses revealed strong correlations between manual and ocular tracking across all measured metrics (maximum correlation, tracking delay, and sensory precision). However, when isolating the specific "multisensory advantage" (the improvement gained from adding sound), these correlations largely disappeared for maximum correlation and tracking delay, while remaining substantial for sensory precision. Furthermore, while general behavioral tracking correlated strongly with underlying sensory noise, this relationship was weaker when analyzing the specific advantage conferred by the added auditory cues.

Significance and Claims
The paper concludes that Continuous Psychophysics is a well-suited tool for investigating multisensory integration. The study specifically highlights the value of combining this behavioral paradigm with Kalman filter analysis to estimate sensory uncertainty directly from behavioral data. By demonstrating that auditory cues reduce estimated sensory noise even when visual information is degraded, the study validates the paradigm's ability to capture the nuanced mechanisms of multisensory processing. The authors modestly frame these results as an expansion of the field initiated by Tonelli et al. (2025), emphasizing the utility of the method for characterizing how the brain integrates conflicting or uncertain sensory inputs in real-time.

Technical Summary: Multisensory Continuous Psychophysics

More like this