Detecting Self-Repairs from Spontaneous Speech with Prompt Ablation Across LLMs and Fine-Tuned Encoder

The human voice carries more than just words; it carries the rhythm of thought. When we speak, especially when we are tired, distracted, or struggling to find the right word, our sentences often stumble. We might start a phrase, stop, and then restart with a different one. In the field of cognitive health, these moments of hesitation and correction are known as self-repairs. For decades, researchers have suspected that the frequency and nature of these self-repairs can serve as a subtle window into the brain's health, potentially signaling early stages of cognitive decline long before more obvious symptoms appear. The challenge has always been that these corrections are fleeting and context-dependent; they are not defined by specific words but by how a speaker changes their mind in the middle of a sentence. Identifying them automatically in a recording is difficult because a computer must understand the flow of conversation, not just the vocabulary.

A team of researchers at the University of Pennsylvania set out to solve this problem by teaching artificial intelligence to spot these self-repairs in spontaneous speech. They focused on a specific type of speech sample where individuals describe the Cookie Theft picture, a standard task used to evaluate memory and language. The goal was to determine if modern large language models—powerful computer systems trained on vast amounts of text—could learn to detect these corrections as accurately as a human expert, and whether smaller, less expensive computer models could do the same job. The researchers compared two very different approaches: using a massive, cloud-based AI that requires significant computing power, and using a much smaller, specialized model that could potentially run on a local device. They also tested whether changing the way they asked the AI to perform the task, such as giving it a specific role or asking it to explain its reasoning, would improve its accuracy.

The study revealed that the size of the AI model mattered significantly, but not in the way one might expect. The largest model, a sophisticated system accessed through a secure internet connection, proved to be the most accurate detector. It correctly identified self-repairs in about 73 percent of the test cases, a performance level that remained steady regardless of how the researchers phrased their instructions. This top-performing model was largely insensitive to the specific details of the prompt, meaning it could reliably find the corrections whether the instructions were simple or complex. In contrast, a smaller, open-source model that was nearly one hundred times less computationally expensive struggled significantly. It managed to detect self-repairs in only about 47 percent of cases and was highly sensitive to the way the instructions were written; changing the prompt could drastically alter its success rate. The researchers found that simply making the instructions more detailed or asking the smaller model to "think" step-by-step before answering did not help it catch up to the larger system. In fact, asking the smaller model to reason through the problem sometimes made it worse, causing it to flag too many non-repairs as corrections.

Perhaps the most surprising finding was that a tiny, specialized computer program, known as a fine-tuned encoder, could perform just as well as the much larger open-source model. This small program, which is designed to be efficient and can run on standard hardware, was trained on a specific dataset of speech corrections. When given enough examples to learn from, it matched the performance of the larger, more expensive open-source model, despite being nearly 100 times smaller. However, it is important to note that while this small model matched the open-source model, both were significantly outperformed by the top-tier cloud-based system. This suggests that for this specific medical task, the bottleneck is not the sheer size of the computer brain, but rather the availability of high-quality, annotated examples of speech corrections. The researchers noted that the small model's performance improved significantly when it was first trained on a general dataset of conversational speech before being specialized for this specific task. This indicates that the limitation lies in the scarcity of labeled data rather than the capacity of the model itself.

The study also uncovered a subtle flaw in how these AI systems handle longer sentences. As the length of a spoken sentence increased, the smaller models began to make more mistakes, falsely identifying normal repetitions or pauses as self-repairs. The larger model did not suffer from this issue as severely, maintaining a steady error rate regardless of sentence length. This finding highlights that while the most powerful models are currently the most reliable, they come with significant drawbacks for clinical use. They require sending patient data to external servers, which raises privacy concerns, and they incur ongoing costs for every use. The smaller, locally deployable models offer a path forward for privacy-preserving tools, but they currently require more training data to reach the same level of reliability.

Ultimately, the research points toward a future where cognitive screening tools can be both accurate and practical. The ability to detect self-repairs automatically is a promising step toward creating a comprehensive system that flags early signs of cognitive impairment. While the most powerful AI systems currently lead in accuracy, the study suggests that with more data and better training, smaller, locally running models could eventually match their performance. This would allow doctors to use these tools in a private office setting without relying on external cloud services, making early detection of cognitive decline more accessible and secure. The work confirms that self-repairs are a viable signal for monitoring brain health, and it outlines a clear path for integrating this detection into routine clinical care, provided the right data resources are developed to train the next generation of efficient, local models.