Investigating Activity-Linked Audio for Conveying Partner Activity Information in Collaborative VR

In the shared spaces of virtual reality, where people meet as digital avatars to build, design, or simply be together, a fundamental challenge remains: how do you know what your partner is doing when you cannot see them? In the physical world, our senses provide a constant, low-level stream of information about the people around us. We hear the shuffle of a footstep, the creak of a chair, or the rustle of clothing, and these sounds tell us where someone is and what they are up to without us needing to look directly at them. This background awareness allows us to work side-by-side without constant eye contact. In a virtual environment, however, these natural sounds are often missing. Users are confined to the narrow field of view provided by their headsets, and the digital avatars they inhabit do not naturally generate the incidental noises of movement. Without these cues, collaborators must constantly turn their heads to check on one another, breaking their focus and disrupting the flow of work. Researchers have long sought ways to fill this sensory gap, exploring whether adding artificial sounds to a virtual world could restore that lost sense of shared presence.

A team of researchers at the University of Tsukuba set out to test this idea by introducing two different types of sound into a collaborative virtual reality task. They wanted to see if hearing a partner's activity through sound would change how people worked together, how they felt about each other, and how they moved their bodies. To do this, they brought twenty-four students into a virtual room to perform a block-assembly task. The participants worked in pairs, taking turns acting as a "Director" and an "Executor." The Director could see a blueprint for a structure and had to verbally instruct the Executor, who had to search four different tables around the room to find the correct blocks and bring them to a central table. The researchers created three different listening environments for these pairs. In one scenario, there was no extra sound at all beyond their voices. In a second, the system played the sound of footsteps whenever a person walked. In the third, the system generated a continuous, abstract sound that grew louder or softer depending on how much the person was moving their entire body, not just their feet. This third sound was designed to act like a sonification, a way of turning general activity levels into a volume of noise that followed the avatar around the room.

The study revealed that adding these sounds did not dramatically change the overall feeling of the experience or the speed at which the pairs completed their task. When the participants were asked afterward how present they felt with their partner or how much they liked the other person, their answers were the same regardless of whether they heard footsteps, abstract noise, or silence. They did not report that one sound made the task easier or the partner more friendly. Furthermore, the researchers found that the sounds did not cause people to turn their heads more often to look at their partner. Even when the partner was working behind them, out of sight, the listeners did not consistently turn their heads in response to the new sounds. This suggests that the sounds did not act as a direct command to look, nor did they fundamentally alter the social atmosphere of the collaboration in a way that the participants could easily describe.

However, a closer look at the data revealed a more subtle and specific change in behavior. The researchers discovered that the sounds did influence how the "Executor" moved, but only in very specific situations. When the Executor was searching for a block at a table located directly behind the Director, their movement patterns changed depending on which sound was playing. In the condition where footsteps were heard, the Executor moved significantly more distance and spent more time moving while they searched compared to the silent condition. In the condition with the abstract noise, they also spent more time moving, though they did not travel as far as those in the footstep condition. This suggests that when the Executor was working in the Director's blind spot, the presence of sound seemed to encourage them to move more, perhaps to make their presence known or to generate the sound themselves. The effect was not a general increase in activity across the whole task; it appeared only when the spatial relationship between the two people made visual contact difficult and the task required searching.

The researchers also listened to the participants' own descriptions of the experience to understand how they interpreted these sounds. Many people described the footsteps as a clear signal that someone was walking or stopping, a direct link between the sound and a specific action. In contrast, the abstract noise was often described as a background hum or an environmental sound, with some participants unsure of exactly what it meant, while others felt it simply told them the partner was active. This difference highlights a trade-off in design: sounds that mimic real-world events are easy to understand but only tell you about that one event, while abstract sounds can cover a wider range of activities but are harder to interpret. The study suggests that in collaborative virtual reality, these activity-linked sounds are not a replacement for visual information, nor are they a magic solution that makes everyone feel closer. Instead, they act as a peripheral cue, a subtle layer of information that becomes useful only when the visual connection is broken and the need to know where a partner is becomes critical. The findings indicate that the value of such sounds depends heavily on the context of the work and the physical relationship between the collaborators, rather than being a constant benefit throughout the entire experience.