💻 computer science

Should We Type or Talk to LLM Agents? A Comprehensive Study of Voice and Keyboard Input Perturbations

This paper introduces HIVE, a perturbation suite demonstrating that voice transcription errors significantly degrade LLM performance by disrupting token structure and hindering reasoning, whereas keyboard typos are more easily absorbed, revealing that preserving original tokens is critical for robustness and that standard training or thinking budgets fail to fully mitigate voice-specific vulnerabilities.

Original authors: Zizhao Hu, Nathan Elijah Segura, Mohammad Rostami, Jesse Thomason

Published 2026-08-05

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zizhao Hu, Nathan Elijah Segura, Mohammad Rostami, Jesse Thomason

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a super-smart robot that can write stories, solve math problems, and write computer code. This robot is called a Large Language Model, or LLM. For a long time, we only talked to these robots by typing on a keyboard, like sending a text message. But now, we can also talk to them out loud, just like talking to a friend. The big question is: does it matter how we talk to them?

When you type, you might make small mistakes, like hitting the wrong letter because your finger slipped, or forgetting to press the space bar. When you speak, the robot has to listen to your voice, turn it into text, and then read it. This process can get messy. Your voice might have "umms" and "ahhs," or the computer might accidentally change a word to a sound-alike word (like hearing "their" instead of "there"). Even worse, some smart tools try to "clean up" your speech before sending it to the robot, rewriting your sentences to make them sound more formal. The scientists wanted to know: do these messy inputs confuse the robot, or is the robot tough enough to handle them? They wanted to find out if it's better to type or to talk, and what kind of mistakes actually break the robot's brain.

To answer this, the researchers built a special testing machine called HIVE (Human Input-Variation Engine). Think of HIVE as a giant "error factory" that takes perfect questions and deliberately messes them up in very specific ways. They created two different kinds of messes: one that looks like a clumsy typist (QWERTY keyboard errors) and one that looks like a messy transcript from a voice recorder (speech errors). They then fed these messy questions to five different smart robots and watched what happened.

Here is what they found, and it's a bit surprising:

1. Speaking is the "Hard Mode" for Robots
When people speak to the robot, the robot gets confused much more often than when people type. But it's not because of the "umms" and "ahhs" (the fillers). The real trouble comes from how the speech is rewritten. If a tool takes your spoken words and reorganizes them into a neat, textbook-style sentence, the robot's performance crashes. In fact, compressing a spoken question into a short, clean summary made the robots 24.1 percentage points less accurate on math problems. It turns out that the robot likes the original structure of your sentence, even if it's a bit messy. Changing the order of your words or rewriting your speech is like taking a map and redrawing the roads; the robot gets lost.

2. Typing Mistakes are Surprisingly Easy to Ignore
On the other hand, the robots are incredibly tough when it comes to typing errors. If you hit the wrong key, swap two letters, or forget a space, the robot usually figures it out. The researchers found that a robot can handle a lot of typing mistakes before it starts to fail. It's only when the mistakes get really concentrated—like messing up almost every word—that the robot gives up. This suggests that robots are very good at guessing what you meant when you type, as long as the core words are still there.

3. The "Token Survival" Rule
The biggest discovery is why one type of mistake hurts more than the other. The researchers found that the robot's brain breaks when the original words (tokens) in your question get destroyed or replaced.

Adding new words (like extra "umms" or long explanations) is cheap for the robot; it doesn't hurt much.
Destroying the original words (by rewriting the sentence or replacing a key letter with a random one) is expensive.
Think of it like a puzzle. If you add a few extra puzzle pieces that don't belong, the robot can still see the picture. But if you take away the pieces that actually form the picture, the robot can't solve it. The "spoken" channel often destroys the original structure of the question, while the "typing" channel usually just adds a little noise without breaking the picture.

4. It Depends on the Task
The gap between typing and speaking only shows up when the robot has to think hard and build an answer from scratch, like solving a math problem or writing code. In these cases, speaking is much worse than typing. However, if the robot just has to pick an answer from a list (like a multiple-choice quiz), it doesn't matter if you type or speak; the performance is the same. The robot needs the clean structure of the original question to do the heavy lifting of reasoning.

5. You Can't Just "Train" the Problem Away
The researchers tried to teach the robots to be better at handling these messy inputs. They tried a technique called "lightweight adaptation," which is like giving the robot a quick crash course on messy speech. It didn't work. The robots couldn't learn to ignore the damage caused by rewriting speech. They also found that this problem isn't just because the robots had seen the answers before (a common issue called "test-set contamination"). Even when they used brand-new math problems the robots had never seen, the spoken channel still performed worse.

6. "Thinking" Helps Typing, But Not Speaking
One of the robots they tested had a special "thinking" mode, where it pauses to think before answering. This "thinking budget" was amazing for fixing typing mistakes. It almost completely recovered the robot's ability to understand messy typed questions. But it did almost nothing to help with spoken questions. If the speech had been rewritten or compressed, the robot's "thinking" couldn't save it. This confirms that the damage from spoken input is too deep; the robot is working with the wrong puzzle pieces, and thinking harder doesn't help.

The Bottom Line
If you want to get the best results from a smart AI, typing is still the safer bet, especially for hard tasks like math or coding. If you must speak, don't rely on a tool to "clean up" your speech or rewrite it for you. The robot actually prefers your raw, slightly messy voice over a polished, rewritten version. And if you are building tools that turn speech into text, the most important advice is: don't restructure what the user said. Just strip out the "umms" and leave the rest alone. The robot is tough enough to handle the mess, as long as you don't break the puzzle.

Technical Summary: Should We Type or Talk to LLM Agents?

Problem Statement

Human interaction with deployed Large Language Models (LLMs) occurs primarily through two channels: typing on a QWERTY keyboard and speaking to a voice interface. Neither channel delivers "clean" input. Typing introduces orthographic noise (e.g., neighbor-key substitutions, transpositions), while speech reaches the model through a transcription layer that introduces disfluencies (fillers, false starts), homophone substitutions, and increasingly, wholesale rewrites by AI-backed dictation tools. While robustness to typographical noise and verbal disfluency has been studied in isolation, there is a lack of systematic comparison between these modalities, particularly regarding the impact of modern AI-driven transcription rewrites. Furthermore, existing benchmarks often rely on synthetic noise that does not reflect the specific motor errors of typing or the structural rewrites of dictation tools.

Methodology: The HIVE Engine

To address these gaps, the authors introduce HIVE (Human Input-Variation Engine), a perturbation suite designed to reproduce the specific errors inherent to human input channels.

Architecture: HIVE consists of 17 operators and 2 controls, implemented via seeded rule-based transforms or few-shot LLM style transfer (using Qwen2.5-7B-Instruct).
Voice Operators (11): These model the transcription pipeline.
- Registers: "Spoken-casual" (podcast-like with high filler rates) and "Spoken-formal" (prepared talk).
- Rewrites: "Spoken-succinct" (compresses and reorders) and "Spoken-clean" (tightens wording while preserving structure).
- Deterministic Factors: Isolate specific surface properties like filler injection, number-to-word conversion, function word removal, casing/punctuation loss, and homophone substitution.
Keyboard Operators (6): These model motor errors on a QWERTY layout.
- Includes neighbor-key substitutions, transpositions, repeated keys, connected-key insertions, missed spaces, and a random replacement control.
- Edits are applied over the Damerau–Levenshtein basis with fixed per-word distances.
Experimental Setup:
- Models: Five instruction-tuned models (Llama-3.1-8B, Qwen2.5-7B, Mistral-7B-v0.3, Qwen3-8B, phi-4).
- Benchmarks: Six datasets covering arithmetic (GSM8K, GSM-Symbolic, GSM1k), code synthesis (HumanEval), and multiple-choice reasoning (MMLU-Pro, TruthfulQA).
- Protocol: Greedy decoding with matched-pair evaluation (clean vs. perturbed) across 5 seeds to isolate perturbation effects from sampling noise.

Key Findings

1. Voice Transcription Costs Accuracy; Structure Matters More Than Fillers

Voice transcription perturbations consistently lower accuracy across all tested instruction-tuned models. The most damaging operator is "spoken-succinct" (compression and reordering), which reduces accuracy by ~24 percentage points. Crucially, the penalty is driven by the structure of the transcription rather than the presence of fillers.

Removing fillers from a spoken transcript ("spoken-filler-stripped") does not recover accuracy.
Adding fillers to a clean text ("clean+fillers") does lower accuracy, but less severely than the full spoken register.
This suggests the penalty is "overdetermined": fillers are sufficient to cause harm, but not necessary, as the register's restructuring (e.g., run-on sentences, clause chaining) independently damages performance.

2. Keyboard Perturbations Are Less Damaging

QWERTY keyboard errors are significantly less costly than voice errors, with an average accuracy drop of ~3.0 points compared to ~9.7 points for voice.

Models absorb a surprising amount of typing error before performance collapses. Accuracy remains flat until roughly 12% of words contain errors, after which it drops sharply.
Concentration vs. Spread: Three edits within 20% of words are twice as damaging as one edit spread across 50% of words, as heavily edited words become unrecoverable in context.
QWERTY-specific errors (neighbor substitutions) are slightly less damaging than arbitrary random replacements, likely because neighbor substitutions often result in less plausible alternative tokens.

3. The Modality Gap is a Reasoning-Task Effect

The performance gap between voice and typing appears almost exclusively in tasks requiring the model to construct or deduce an answer (e.g., arithmetic, code generation).

On multiple-choice tasks where the answer is selected from provided options, the gap between voice and typing is negligible.
On generation tasks, voice inputs result in 6–7 percentage points lower accuracy than typing.

4. Token Survival is the Primary Predictor of Harm

The fundamental cause of performance degradation is the destruction of original tokens.

There is a strong positive correlation ( $r=+0.79$ ) between the percentage of original question tokens that survive the perturbation and the resulting accuracy.
Adding new tokens (token inflation) has a minimal negative impact compared to destroying existing ones.
Operators that preserve the original token sequence (even with added fillers) are far less damaging than those that rewrite or compress the text, which destroys the original tokenization.

5. Harm is Not Solely Due to Test-Set Contamination

The performance penalty persists on decontaminated datasets (GSM-Symbolic and GSM1k), which regenerate problems with new surface forms while preserving reasoning logic. This indicates that the harm is not merely due to models retrieving memorized solutions from training data but reflects a genuine inability to process the perturbed input structure.

6. Lightweight Adaptation Cannot Mitigate the Harm

Attempts to train models to absorb these perturbations via self-distillation (LoRA adapters) were largely unsuccessful.

High-rank adaptation settings could close the gap for conversational speech, but only by dragging down the model's performance on clean inputs, which violates the constraint that robustness should not come at the cost of base performance.
Low-rank settings preserved clean accuracy but failed to improve performance on perturbed inputs.
Adding terms to penalize attention to fillers or align hidden states did not close the gap. The harm appears intrinsic to the model's current architecture and training distribution.

7. "Thinking" Budgets Recover Keyboard but Not Voice

Enabling a "thinking" mode (chain-of-thought reasoning) in Qwen3-8B almost entirely recovers performance on keyboard perturbations but leaves spoken registers essentially untouched.

Reasoning helps reconstruct garbled text where original tokens are preserved (keyboard errors).
It fails to recover performance when the input structure is fundamentally altered or compressed (voice rewrites); notably, compressed speech performance actually gets worse with a thinking budget enabled.
This suggests that reasoning can repair token-level noise but cannot compensate for the loss of the original token sequence and structural integrity inherent in voice rewrites.

Significance and Claims

The paper claims that speaking is currently the "expensive channel" for interacting with LLMs, primarily due to the structural rewrites introduced by AI-powered dictation tools. The authors argue that the damage is not a simple matter of noise but a loss of the original token sequence and structural integrity required for reasoning.

Practical Implications:

For Users: When using voice interfaces, the phrasing spoken is the phrasing the model must work from; users should avoid rambling or relying on downstream tools to "clean" the text, as minimal disfluency stripping is preferable to structural rewriting.
For Developers: Dictation front-ends should minimize reformatting and restructuring of user input. The most significant harm measured comes from the "rewrite" layer of dictation tools, which is also the "cheapest thing to change."
For Research: Robustness to input perturbation is modality-specific. Models robust to keyboard typos are not necessarily robust to spoken disfluency, and lightweight adaptation is insufficient to bridge this gap.

The study concludes that while typing errors are absorbable by reasoning models, the structural alterations inherent in voice transcription (particularly AI-driven rewrites) pose a fundamental challenge that current models and lightweight adaptation strategies cannot fully overcome.