Hierarchical Uncertainty Transformation During Multimodal Comprehension

Human communication is rarely a simple stream of words. In a face-to-face conversation, meaning arrives through a tangled web of spoken language and hand gestures, each moving at its own speed and carrying its own weight. Sometimes a gesture clarifies a vague word; other times, a spoken phrase makes sense of a confusing motion. The brain must weave these separate threads into a single, coherent understanding, even when the information from each source is incomplete or ambiguous. For decades, scientists have understood that the brain treats this process as a problem of probability: it holds multiple possible meanings in mind at once, weighing how likely each one is, and then settles on the best fit as more evidence arrives. The prevailing view has been that as we receive more information, the brain simply reduces the amount of uncertainty, narrowing down the possibilities until the answer becomes clear.

A new study challenges this simple idea of steady reduction. Researchers at the Institute of Psychology, Chinese Academy of Sciences, propose that the brain does not just lower the volume of uncertainty; it actually changes the very nature of that uncertainty as it moves from one computational level to the next. By watching how people interpret gestures and speech fragments in real time, the team discovered that uncertainty transforms in three distinct, interconnected ways. It starts as confusion within a single sense, shifts as the senses talk to each other, and finally settles into a new form when the brain merges them into a single meaning. This process is not a fixed sequence of steps but a flexible system that reorganizes itself depending on how much information is available from each source.

To uncover this hidden architecture, the researchers designed a series of experiments where they could control exactly how much meaning was available in a gesture versus a speech fragment. They recorded videos of people making hand gestures and audio of them speaking verbs. In preliminary tests, they determined the precise moment in time when a gesture or a word became clear enough for a person to identify its meaning. They then cut the videos and audio into three types of fragments: pieces that were too short to be understood, pieces that were just long enough to be identified, and pieces that were longer and fully clear. By mixing and matching these fragments, they created scenarios where both signals were unclear, where one was clear and the other was not, and where both were clear.

In a behavioral experiment, participants were asked to name what they saw and heard. The researchers did not just look at whether the participants got the answer right; they analyzed the distribution of responses. For each stimulus condition, the naming responses were aggregated across participants; a probability distribution was estimated from the relative frequencies of the reported verbs, and Shannon entropy was calculated from that distribution. Using this mathematical approach based on information theory, the team measured this uncertainty not as a single number, but as a shifting landscape of possibilities. They found that uncertainty behaves differently depending on which computational level of the brain's processing it is in.

One computational level involves the uncertainty within each sense on its own. The researchers found that as a gesture or a speech fragment became longer and clearer, the number of possible meanings dropped, and the brain's uncertainty decreased. This is the familiar process of narrowing down options. However, the story changed when the two senses were presented together. When a clear speech signal was paired with a vague gesture, the brain did not just ignore the vague gesture; it actively reshaped the uncertainty of that gesture based on the speech. Conversely, a clear gesture could reshape the uncertainty of a vague word. Crucially, the effects of each modality on the interpretation of the other were asymmetric and depended on the semantic information available in each modality. This showed that the interaction between senses is not a simple averaging of information but a complex, directional transformation.

Another computational level occurred when the brain fused the two signals into a single, unified meaning. Here, the researchers found that the uncertainty of the final result could not be explained simply by adding up the changes from the previous levels. Instead, the brain created a new kind of uncertainty that belonged to the combined system. To see how the brain handled these levels, the team recorded electrical activity from the participants' brains using sensors on the scalp. In this main EEG experiment, participants judged the speaker's gender, which allowed the researchers to examine implicit gesture-speech semantic integration without requiring explicit semantic judgments. They looked for specific patterns of brain waves that corresponded to the different types of uncertainty they had calculated.

The results revealed that the neural associations for these different levels were distinguishable but partially overlapping, and their timing varied across semantic configurations. Modality-specific uncertainty was associated with activity in the earlier N1-P2 range, though these associations were not confined to early time windows. The bidirectional cross-modal uncertainty transformation—where gesture and speech gains reshaped one another—was also associated with activity in the earlier N1-P2 range, with additional associations in the N400 range depending on the modality and semantic configuration. The N400, an event-related potential component sensitive to context-dependent semantic processing, including the integration of meaning across speech and gesture, was also involved. Finally, multisensory convergence, indexed by aligned fusion entropy and unit gain, showed particularly consistent associations in the N400 range.

Perhaps most importantly, the study showed that this entire system is dynamic. The way the brain expressed these different types of uncertainty changed depending on the specific combination of clear and unclear signals. When both the gesture and the speech were vague, the brain's response looked different than when one was clear and the other was not. The neural signals did not follow a rigid, unchanging script. Instead, the brain reorganized its activity in real time to match the specific demands of the situation. This flexibility indicates that the brain is not just passively waiting for information to arrive and then reducing uncertainty; it is actively restructuring how it handles uncertainty as the conversation unfolds.

The researchers also used statistical models to show that these computational levels are deeply connected. The way the brain transformed uncertainty in the cross-modal level was directly linked to how it converged the signals in the multisensory convergence level. This connection was not fixed; it shifted depending on whether the gesture or the speech was the clearer signal. This finding supports the idea that the brain operates as a unified, hierarchical system where different levels of processing are constantly influencing one another. The findings suggest that multimodal comprehension involves transformations in the computational form of uncertainty, alongside changes in its magnitude as evidence accumulates.

This work provides a new framework for understanding how we make sense of the world when information comes from multiple sources. It suggests that the brain's ability to understand complex communication relies on its capacity to transform uncertainty, not just reduce it. By recognizing that uncertainty changes its nature at different computational levels, scientists can better understand how the brain builds coherent reality from fragmented and ambiguous signals. The findings offer a glimpse into the intricate machinery of human communication, revealing that the path to understanding is not a straight line of decreasing doubt, but a dynamic journey of reorganization and adaptation.