TellTale: Blending Multi-Instance LoRA Text Encoders and a Zero-Shot LLM Judge for Ambivalence/Hesitancy Recognition in Videos
TellTale is a text-only approach for recognizing ambivalence and hesitancy in interview videos that achieves state-of-the-art performance by combining fine-tuned multi-instance LoRA text encoders with a zero-shot LLM judge, significantly outperforming vision-based baselines on the BAH dataset.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to guess how a person is feeling just by listening to them talk. Sometimes, people say one thing but mean another, or they stumble over their words because they are unsure, conflicted, or hesitant. In the world of artificial intelligence, this is a tricky puzzle called "ambivalence" or "hesitancy" recognition. Scientists have been building computer programs to spot these mixed feelings, hoping to help with things like digital health coaching, where a hesitant answer might mean a patient isn't ready to change a habit yet. Usually, these programs try to be like super-observers: they watch the person's face, listen to their voice, and read their words all at once. But what if the most important clues are hidden entirely in the words themselves, and looking at the face actually distracts the computer? That is the big question this new research tackles.
The paper introduces a clever system called "TellTale," which acts like a detective that refuses to look at the suspect's face or listen to their tone of voice. Instead, it focuses exclusively on the transcript—the written record of what was said. The researchers tested this idea on a dataset of interview videos where people were asked questions, and the goal was to figure out if the person was showing signs of hesitation or conflict. While other systems tried to analyze the video and audio, TellTale decided to ignore them completely. It turns out that for this specific task, the words alone were the golden ticket. The system achieved a score of 0.7364 (a measure of how often it was right), which is a massive jump compared to the official "vision-based" system that tried to watch the video and only scored 0.2827.
So, how does TellTale solve the mystery? Think of it as a panel of three different experts, each reading the same script but using a different superpower to find the hesitation.
First, there are two "Text Encoders." Imagine these as two very smart, well-read librarians who have been trained specifically to spot the subtle signs of doubt in a story. They don't read the whole story at once; instead, they chop the transcript into tiny 3-to-5-second chunks, like slicing a loaf of bread. They look at each slice and ask, "Does this little piece sound hesitant?" The trick is that they don't just average the answers. If a person talks confidently for most of the video but stumbles and hedges for just one tiny moment, the system knows that one moment is the key. It uses a "smooth maximum" method, which is like a spotlight that shines brightest on the single most hesitant chunk of text, ignoring the boring, confident parts. These two librarians are slightly different: one is a multilingual expert, and the other is a specialist in a different style of reading. Because they are different, they make different mistakes, which helps the team catch more errors.
The third expert is a "Zero-Shot LLM Judge." This is like a brilliant, well-read judge who has never been trained on this specific test before. You simply hand them the transcript and say, "On a scale of 0 to 100, how hesitant does this person sound?" The judge gives an answer based on their general knowledge of human language and behavior. This judge is special because they haven't been "trained" on the specific answers of the other librarians, so they bring a completely fresh perspective. If the librarians get confused by a tricky sentence, the judge might see it clearly, and vice versa.
The final step is the "Blend." The system takes the scores from the two librarians and the judge and mixes them together like a recipe. It gives the most weight (45%) to the strongest librarian, a solid chunk (30%) to the second librarian, and a significant portion (25%) to the judge. Then, it draws a line in the sand: if the final mixed score is above 0.53, the system decides, "Yes, this person is hesitant."
The researchers found that this text-only approach was incredibly effective. By ignoring the video and audio, they avoided getting distracted by things that didn't actually help. The combination of the two trained librarians and the fresh-minded judge created a team that was much stronger than any single member. On the final test with 152 videos from people the system had never seen before, TellTale got it right about 74% of the time (specifically, a Macro-F1 of 0.7364). This was a huge improvement over the vision-based system, which struggled to get it right even 30% of the time.
The paper suggests that for this specific type of interview, the hesitation is mostly in the words, not the face or the voice. The system proved that you don't need a camera or a microphone to spot a conflicted mind; you just need to listen closely to what is being said. The method is also designed to be simple and cheap to run, using small, efficient upgrades to the computer's brain rather than requiring massive, expensive supercomputers. While the researchers note that visual cues might still exist, their experiments showed that for this task, focusing purely on the transcript was the winning strategy.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.