💬 NLP

When Silence Matters: The Impact of Irrelevant Audio on Text Reasoning in Large Audio-Language Models

This paper demonstrates that irrelevant audio inputs, including silence and noise, significantly degrade the text reasoning performance and stability of Large Audio-Language Models, revealing a critical cross-modal interference challenge that existing mitigation strategies only partially address.

Original authors: Chen-An Li, Tzu-Han Lin, Hung-yi Lee

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chen-An Li, Tzu-Han Lin, Hung-yi Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a brilliant detective trying to solve a math puzzle written on a whiteboard. You are perfectly capable of solving it just by reading the words. But now, imagine someone stands next to you and starts playing a radio.

In this paper, researchers asked a simple question: What happens if the radio is playing static, silence, or the sound of rain, while you try to solve the puzzle?

Intuitively, you might think, "It doesn't matter! I'm just reading the text. The noise is irrelevant." You'd expect the noise to be ignored, like background chatter in a coffee shop.

The shocking discovery is that it does matter. Even if the audio makes no sense and has nothing to do with the math problem, it actually messes up the detective's brain.

Here is a breakdown of their findings using everyday analogies:

1. The "Silence" Trap

We often think silence is neutral—like an empty room. But for these AI models (called Large Audio-Language Models), silence is just as distracting as loud noise.

The Analogy: Imagine trying to read a book in a library. If someone whispers "shhh" repeatedly, it's annoying. But if they just stand there holding their breath, staring at you, it can be more distracting because your brain is waiting for a sound that never comes. The AI gets confused by the "empty" audio just as much as by loud static.

2. The "Volume" and "Duration" Effect

The researchers found that the longer the noise lasts and the louder it is, the worse the AI performs.

The Analogy: If a fly buzzes around your head for one second, you might ignore it. If it buzzes for 30 minutes, you can't focus. Similarly, if the background noise is a whisper, the AI might cope. If it's a shout, the AI's "brain" starts to scramble, and it makes mistakes on simple math problems it could have solved easily.

3. The "Temperature" Factor

In AI, "temperature" is like a dial that controls how creative or random the model's answers are.

The Analogy: Think of the AI as a student taking a test.
- Low Temperature: The student is serious, focused, and sticks to the facts. They are less likely to be distracted by the noise.
- High Temperature: The student is daydreaming, taking risks, and being creative. When you add noise to this daydreaming student, they get completely lost. The noise amplifies their confusion, making them flip-flop between right and wrong answers.

4. Why Does This Happen?

These models are designed to listen to audio and read text at the same time. They try to "mix" the two signals together to understand the world.

The Analogy: Imagine a chef trying to taste a soup. If the chef is also listening to a radio, the radio waves might interfere with the chef's ability to taste the salt. Even if the radio is playing a song about soup, it shouldn't change the taste. But in these AI models, the "mixing bowl" is so sensitive that even empty air (silence) or random static changes the flavor of the final answer.

5. Can We Fix It?

The researchers tried two simple tricks to stop the AI from getting distracted:

Trick #1: The "Please Focus" Reminder (Prompting)
- What they did: They told the AI, "Hey, ignore the noise and just look at the text."
- The Result: It barely worked. It's like telling a distracted driver, "Please look at the road," while a siren is blaring right next to them. The instruction wasn't strong enough to override the confusion.
Trick #2: The "Ask Three Friends" Strategy (Self-Consistency)
- What they did: Instead of asking the AI for one answer, they asked it the same question eight times and took the most common answer.
- The Result: This worked! It was like asking a group of friends to solve the puzzle. Even if one friend got distracted by the noise, the majority would still get it right.
- The Catch: It takes a lot more time and computer power to ask the same question eight times. It's effective, but it's expensive and slow.

The Big Takeaway

This paper warns us that AI isn't as robust as we think. Just because a model is smart at reading and listening doesn't mean it can ignore garbage data.

If you build a real-world app that uses these models (like a voice assistant helping you with homework), you need to be careful. Even if the user is silent or there is background noise, the AI might start giving you wrong answers or changing its mind randomly.

The lesson: We need to teach these AI models how to "tune out" the noise better, rather than just hoping they will ignore it on their own.

1. Problem Statement

Large Audio-Language Models (LALMs) are designed to unify speech and text processing. However, current evaluations often assume clean, modality-aligned inputs. In real-world deployment, text-only reasoning tasks frequently encounter "irrelevant audio" (e.g., silence, background noise, environmental sounds) alongside the text input.

The core research question is: To what extent does non-informative audio interfere with the text reasoning capabilities of LALMs? The authors challenge the intuition that silence or irrelevant noise should be neutral, hypothesizing that these signals may disrupt the cross-modal fusion process, leading to degraded performance and unstable predictions.

2. Methodology

Experimental Setup

Benchmarks: The study evaluates three text-based reasoning benchmarks where audio is unnecessary:
- GSM8K: Arithmetic reasoning.
- ARC-Challenge: Science question answering.
- MMLU: Massive multi-task language understanding.
Models: A diverse set of state-of-the-art open-source LALMs was tested, ranging from 3B to 24B parameters (e.g., Qwen2.5-Omni, Phi-4-Multimodal, Voxtral, DeSTA2.5-Audio).
Interference Types: The audio channel was varied while keeping text fixed, introducing three types of non-semantic perturbations:
1. Silence: 5 seconds of silence.
2. Synthetic Noise: 5 seconds of Gaussian noise at -40 dBFS.
3. Environmental Sounds: Real-world samples from the FSD50K dataset (rain, animals, etc.).
Variables: The study systematically ablated:
- Duration: 1s, 5s, 10s, 30s.
- Amplitude: -60 dBFS, -40 dBFS, -20 dBFS.
- Decoding Temperature: 0.0 (greedy) to 1.0.

Evaluation Metrics

Accuracy (Acc): The proportion of correctly answered samples.
Influence Rate (IR): A novel metric adapted from multimodal robustness studies to measure instability. It calculates the frequency with which the model's prediction flips (from correct to incorrect or vice versa) when irrelevant audio is introduced, regardless of whether the flip improves or harms the final score.
$IR = \frac{n_{ic} + n_{ci}}{N}$

3. Key Results

Cross-Modal Interference is Pervasive

Performance Degradation: All evaluated LALMs showed reduced accuracy when irrelevant audio was present, even for tasks solvable by text alone.
The "Silence" Paradox: Contrary to expectations, silence destabilized model outputs as strongly as synthetic noise. Persistent silence is not ignored by the model but actively disrupts the fusion mechanism.
Instability: The Influence Rate was significantly higher under interference conditions, indicating that models frequently changed their reasoning path or final answer due to the presence of non-informative audio.

Scaling Effects

Model Size: Larger models (e.g., 24B) demonstrated greater resilience than smaller ones (e.g., 3B), showing smaller accuracy drops and lower Influence Rates. However, no model was fully robust; vulnerabilities persisted across all sizes.
Architecture: Factors beyond parameter count (e.g., training data, optimization design) influenced robustness, as some models of similar size exhibited different volatility levels.

Sensitivity to Conditions

Duration & Amplitude: Interference severity scales linearly with the duration of the audio and the amplitude (loudness) of the noise. Longer silence or louder noise causes greater degradation.
Decoding Temperature: Higher decoding temperatures significantly amplify the negative effects of irrelevant audio. While low temperatures (greedy decoding) offer some stability, stochastic sampling makes models highly volatile in the presence of noise.
Task Dependency: Complex, broad-domain tasks (MMLU) suffered more severe degradation than structured tasks (GSM8K).

4. Mitigation Strategies

The authors tested two straightforward mitigation approaches:

Prompting: Prepending an instruction like "Focus on the text or audio that contains useful information."
- Result: Ineffective. The prompt yielded inconsistent results, sometimes slightly lowering the Influence Rate but often reducing accuracy or amplifying instability. It failed to systematically counteract cross-modal interference.
Self-Consistency: Generating multiple responses (8 samples) at a moderate temperature and selecting the majority vote.
- Result: Effective but Costly. This approach significantly improved accuracy and reduced the Influence Rate across all interference types. However, it comes at the cost of a substantial increase in computational overhead (test-time compute).

5. Key Contributions & Significance

Discovery of "Silence" as a Distractor: The paper establishes that silence is not a neutral input in LALMs; it acts as a disruptive signal comparable to noise, challenging the assumption that non-speech audio is harmless.
Quantification of Cross-Modal Interference: By introducing the Influence Rate, the authors provide a metric to capture model instability that accuracy alone misses. This reveals that models are "confused" by irrelevant inputs even when they occasionally guess correctly.
Robustness Benchmarking: The study provides a systematic evaluation framework for LALMs under realistic, noisy conditions, highlighting that current models lack the necessary robustness for real-world deployment where audio streams are often uncurated.
Implications for Fusion Strategies: The findings suggest that current fusion mechanisms in LALMs are too sensitive to non-semantic inputs. Future work must focus on developing efficient fusion strategies that can filter or ignore irrelevant modalities without relying on expensive inference-time techniques like self-consistency.

Conclusion

The paper concludes that irrelevant audio is a critical, under-explored vulnerability in Large Audio-Language Models. It degrades reasoning performance, increases prediction volatility, and scales with signal duration and intensity. While larger models offer marginal improvements, the field requires new architectural solutions to ensure robust reasoning in the presence of multimodal noise.