💬 NLP

SMILE-Next: Teaching Large Language Models to Detect, Classify, and Reason about Laughter

This paper introduces SMILE-Next, a comprehensive real-world laughter dataset, and proposes a specialized large language model framework featuring Laughter-specific Self-Instruct and a Mixture-of-Laugh-Experts (MoLE) mechanism to significantly improve the detection, classification, and reasoning of complex social laughter signals.

Original authors: Lee Jung-Mok, Kim Sung-Bin, Joohyun Chang, Lee Hyun, Tae-Hyun Oh

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lee Jung-Mok, Kim Sung-Bin, Joohyun Chang, Lee Hyun, Tae-Hyun Oh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine laughter as a complex, secret code that humans use to communicate. It's not just about finding something funny; sometimes we laugh to be polite, to hide awkwardness, or to show we're nervous. For a long time, computers have been terrible at cracking this code. They could tell that someone laughed, but they struggled to understand why or what kind of laugh it was.

This paper introduces SMILE-Next, a new project designed to teach computers to become "laughter detectives." Here is how they did it, explained simply:

1. The New "Textbook" (The Dataset)

Think of the researchers as creating a massive, new textbook for AI. Before this, the textbooks only had a few pages on laughter, mostly from TV shows or TED talks.

What's new: They collected thousands of video clips from real life—talk shows, movies, and even two people chatting on the street.
The Three Lessons: Instead of just asking "Did they laugh?", the textbook teaches the AI three specific skills:
1. Detection: Spotting that a laugh happened.
2. Classification: Identifying the type of laugh (Is it a happy "mirthful" laugh? A polite "I'm nodding along" laugh? Or an awkward "I don't know what to say" laugh?).
3. Reasoning: Explaining why it happened. (e.g., "He laughed because his boss made a joke, but he was actually nervous about the meeting.")

2. The "Translator" Strategy (Textualization)

This is the paper's biggest trick. Usually, AI tries to watch a video and listen to audio all at once, like a person trying to read a book while listening to a loud radio. It gets confused because the signals are all tangled together.

The researchers decided to translate everything into text first.

The Metaphor: Imagine the video is a foreign language. Instead of forcing the AI to learn the language from scratch, they hired a team of translators (specialized tools) to turn the video into a written story.
The Story includes: What was said (transcript), how the voice sounded (pitch, tone), what the faces looked like (smiles, frowns), and who the people were to each other (boss/employee, friends).
Why it works: By turning the video into a story, the AI can use its superpower—reading and understanding language—to figure out the joke. It's much easier for a computer to "read" a description of an awkward silence than to "watch" one.

3. The "Specialized Team" (Mixture-of-Laugh-Experts)

Once the AI has the "textbook" and the "translated stories," the researchers needed a way to teach it to be good at all three lessons (detecting, classifying, reasoning) without getting confused.

The Metaphor: Imagine a general doctor who is good at everything but not a specialist. The researchers gave this doctor a team of three specialized assistants (called "Experts").
- Expert 1 is great at spotting the laugh.
- Expert 2 is great at guessing the type of laugh.
- Expert 3 is great at explaining the reason.
The Router: There is a "manager" (a router) that looks at the question. If the question is "Did they laugh?", the manager calls Expert 1. If the question is "Why did they laugh?", it calls Expert 3. They all work together in the same brain, but they switch roles depending on the job. This makes the AI faster and smarter.

4. The "Practice Drills" (Self-Instruction)

The real-world videos they collected were great, but not enough to cover every possible situation.

The Metaphor: To make the AI smarter, they used a powerful AI (GPT-4) to write new practice drills.
The AI was told: "Here are some examples of laughs. Now, invent 1,000 new scenarios where people might laugh, including weird or awkward ones."
This allowed the AI to practice on situations it had never seen before, making it much better at handling real-life surprises.

The Results

When they tested this new system:

It was much better at understanding laughter than previous models that tried to watch videos directly.
It could correctly identify that a laugh was "awkward" rather than "funny" by looking at the body language and tone described in the text.
Humans preferred its explanations over other models because they felt more accurate and human-like.

In short: The paper didn't just build a better laugh detector; it built a system that translates video into a story, uses a team of specialists to analyze that story, and practices on made-up scenarios to become a master of human social cues.

Technical Summary: SMILE-Next

Problem Statement

Laughter is a complex social signal that conveys communicative intent beyond simple amusement, serving functions such as building intimacy, capturing attention, and easing tension. While prior research has explored laughter and humor, existing approaches are largely limited to narrowly defined tasks (e.g., humor detection, intensity estimation) or constrained, scripted scenarios (e.g., TED talks, sitcoms). These methods struggle to capture laughter as it naturally occurs in diverse real-world human-human interactions. Furthermore, current multimodal models often rely on implicit feature fusion, which entangles causal, social, and contextual cues, making it difficult to provide broad, context-aware explanations for why laughter occurs. There is a lack of comprehensive datasets and specialized models capable of detecting, classifying, and reasoning about laughter across varied social contexts.

Methodology

The authors propose SMILE-Next, a comprehensive dataset and a specialized Large Language Model (LLM) framework designed for real-world laughter understanding.

1. The SMILE-Next Dataset

SMILE-Next is constructed to support generalized laughter reasoning through three core tasks:

Laughter Detection: A binary classification task to determine if a video clip induces laughter.
Laughter Type Classification: A multi-class task categorizing laughter into three types: mirthful (spontaneous/joyful), polite (socially motivated/non-spontaneous), and satirical (sarcastic/awkward/mocking).
Laughter Reasoning: A free-form text generation task explaining the underlying cause of laughter.

Construction Pipeline:

Data Collection: Videos are sourced from diverse domains including TED Talks, sitcoms, and "in-the-wild" YouTube content (talk shows, dyadic conversations, movies).
Textualization: To disentangle entangled multimodal signals, the authors convert video and speech attributes into structured textual representations. This includes extracting utterances, facial action units, acoustic features (pitch, intensity, jitter, shimmer), video captions, and speaker relationships.
Annotation: A hybrid approach is used where an expert LLM (GPT-4) generates pseudo-labels for reasoning, which are then verified and refined by human annotators via Amazon Mechanical Turk (AMT). The final dataset contains 3,590 video clips and 6,386 question-answer pairs.

2. The Laughter-Specialized LLM Framework

Building on SMILE-Next, the authors propose a unified framework for a laughter expert LLM consisting of two key components:

Laughter-Specific Self-Instruct: To overcome the scale and diversity limitations of curated data, the authors employ a self-instruction framework. Using an external LLM (GPT-4), they automatically synthesize diverse instruction-response pairs based on the seed tasks. This expands the dataset to cover a broader range of social contexts (e.g., workplace hierarchies, informal friendships) and laughter types (e.g., nervous, forced, sarcastic), enhancing the model's generalization capabilities.
Mixture-of-Laugh-Experts (MoLE): To handle multiple laughter-related tasks within a unified model without catastrophic forgetting, the authors introduce a parameter-efficient MoLE framework.
- The backbone LLM weights are frozen to preserve general reasoning knowledge.
- Task-specific lightweight experts are implemented using LoRA (Low-Rank Adaptation) adapters injected into the linear layers.
- A Router Gating Network dynamically computes weights to select and combine the outputs of these experts ( $E_1, E_2, E_3$ ) for each input instance. This allows the model to adaptively specialize in detection, classification, or reasoning while sharing a common backbone.

Key Contributions

SMILE-Next Dataset: The introduction of a large-scale, multimodal dataset covering diverse real-world interaction scenarios with three complementary tasks (detection, classification, reasoning) and rich textualized annotations.
Laughter Expert LLM: The development of a specialized LLM that demonstrates strong capabilities in identifying and interpreting laughter in social interactions.
Novel Training Strategies: The proposal of Laughter-specific Self-Instruction to scale data diversity and the Mixture-of-Laugh-Experts (MoLE) framework to efficiently handle multi-task learning with parameter efficiency.
Textualization Strategy: Empirical validation that converting multimodal signals into explicit textual representations enables LLMs to better leverage their pre-trained commonsense and causal reasoning capabilities compared to models processing raw implicit multimodal features.

Experimental Results

The proposed approach was evaluated on SMILE-Next against various baselines, including Audio-Visual LLMs (e.g., MiniCPM-o, Qwen-Omni) and Visual LLMs (e.g., Video-LLaVA, Qwen2.5-VL).

Quantitative Performance: The proposed LLMs (Vicuna, LLaMA3, Qwen2.5) trained with Self-Instruction and MoLE significantly outperformed all baselines across all three tasks. For instance, in laughter detection, the proposed approach achieved an F1 score of 0.9675 (Vicuna) compared to 0.1952 for MiniCPM-o and 0.8586 for Qwen2.5-VL. Similar gains were observed in classification and reasoning metrics (BLEU, METEOR, ROUGE, SentBERT).
Ablation Studies:
- Multimodal Cues: Utilizing the full set of textualized cues (transcription, acoustic, visual, and relational) consistently outperformed models using only text transcripts, confirming the necessity of disentangled multimodal information.
- Self-Instruction & MoLE: Both components contributed to performance gains. Self-instruction improved generalization to diverse scenarios, while MoLE further enhanced task-specific specialization.
- Router Analysis: Visualization of the gating weights revealed that different tasks activate different experts (e.g., Expert 1 dominates reasoning, while Experts 2 and 3 contribute more to classification and detection), confirming effective task-adaptive routing.
Human Preference: In a human evaluation, the proposed LLM's reasoning outputs were preferred over those of AV-LLMs and V-LLMs in 69.0% of comparisons against AV-LLMs and 55.7% against V-LLMs, indicating better alignment with human judgment.
Efficiency: The MoLE framework introduced negligible latency (approx. 1-2% increase) compared to single-expert models.

Significance and Claims

The paper claims that SMILE-Next and the associated framework advance the state of the art in real-world laughter understanding by moving beyond simple detection to nuanced reasoning. The authors emphasize that textualizing multimodal information is essential for robust laughter understanding, as it allows LLMs to explicitly process social and affective factors that are often entangled in latent feature spaces. By combining self-instruction for data diversity and MoLE for task specialization, the proposed system achieves a balance between generalization and specific expertise, enabling machines to interpret the complex, context-dependent nature of human laughter more effectively than previous multimodal approaches. The work is presented as a step toward richer human-machine and machine-machine social interactions.