💻 computer science

Evaluating Large language models on Understanding Korean indirect Speech acts

This study evaluates the ability of various large language models to understand Korean indirect speech acts by constructing a specialized dataset and employing both automated and human assessments, revealing that while proprietary models like Claude3-Opus outperform open-source alternatives, none yet match human-level proficiency in interpreting context-dependent intentions.

Original authors: Youngeun Koo, Jiwoo Lee, Dojun Park, Seohyun Park, Sungeun Lee

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Youngeun Koo, Jiwoo Lee, Dojun Park, Seohyun Park, Sungeun Lee

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are chatting with a friend, and they say, "It's freezing in here." If you are standing next to a broken air conditioner, they are likely just stating a fact. But if you are sitting in a warm living room with the windows closed, they are probably hinting, "Please close the window" or "Turn up the heat." This gap between what words literally say and what they actually mean is the secret sauce of human conversation. In the world of science, this is called pragmatics: the study of how context changes meaning. While computers have gotten incredibly good at translating languages or solving math problems, they often struggle with this "reading between the lines" skill. They might take your hint literally and just say, "Yes, the temperature is low," missing the point entirely. This matters because as we start relying on AI assistants for everything from scheduling meetings to giving advice, we need them to understand not just our words, but our intentions.

This paper is like a detective story where researchers put a bunch of Artificial Intelligence (AI) models to the test to see if they can crack the code of indirect hints. The scientists focused on Korean, a language where context is king, and they built a special quiz based on a classic theory of how humans use language. They created 240 scenarios where the same sentence could mean two very different things depending on the situation. For example, saying "The ice is thin" could be a simple fact, or it could be a warning to kids not to step on a frozen pond. The researchers wanted to see if AI could tell the difference between a direct statement and an indirect request, promise, or expression of emotion.

The results were a mix of "not bad" and "still a long way to go." The researchers tested 12 different AI models, ranging from massive, expensive systems to smaller, open-source ones. The clear winner was a model called Claude3-Opus. It scored about 71.94% on a multiple-choice test and 65% on a more open-ended test where it had to explain its reasoning. While this is impressive, it's important to note that even the best AI model didn't beat the humans. The human participants in the study scored around 77.64%, proving that people are still the masters of understanding subtle social cues.

The study found that most AI models are great at taking things literally but terrible at guessing the hidden meaning. If a sentence was a direct request, the AI got it right most of the time. But when the request was indirect—like saying "It's nice outside" to suggest going for a walk instead of just talking about the weather—most models stumbled. They tended to get stuck on the literal meaning of the words. Interestingly, the AI models were surprisingly good at understanding indirect compliments or idioms (like saying someone is a "chef" because they cooked a great meal), but they struggled the most with indirect requests.

The researchers also noticed some funny quirks in how the AI behaved. Some smaller models, when asked to answer in Korean, accidentally replied in English, as if they were confused about who they were talking to. Others tried to force multiple-choice answers even when the question asked for a free-form explanation, as if they were trying to game a test they had seen before. The paper suggests that while these AI models are powerful, they still lack the deep, intuitive grasp of human context that even a teenager has. They are like students who have memorized the dictionary but haven't yet learned how to have a real conversation. The study concludes that to make AI truly helpful, we need to teach them to look past the surface of words and understand the messy, context-filled world of human intention.

Technical Summary: Evaluating Large Language Models on Understanding Korean Indirect Speech Acts

Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable proficiency in traditional Natural Language Processing (NLP) tasks such as categorization, translation, and logical reasoning, their ability to comprehend human intentions in conversational contexts remains under-evaluated. Current benchmarks predominantly focus on knowledge processing and explicit literal meanings, often neglecting pragmatics—the study of language use within specific contexts.

A critical gap exists in evaluating how well LLMs understand indirect speech acts, where the speaker's actual intention differs from the literal meaning of the utterance. This is particularly challenging in Korean, a high-context language where shared background information and situational cues are essential for effective communication. Existing evaluations rarely address the nuance of distinguishing between direct and indirect intentions based on conversational context, a capability crucial for LLMs acting as personal agents in daily life.

Methodology

Dataset Construction

The authors constructed a specialized evaluation dataset consisting of 240 items in Korean, grounded in Searle's Speech Act Theory. The dataset categorizes speech acts into five types (Representatives, Directives, Commissives, Expressives, Declaratives) and focuses on three specific types of indirect speech acts where a declarative sentence (Representative) performs a different illocutionary act:

ReDi (Representative $\to$ Directive): A factual statement used to suggest or request an action (e.g., "It's nice outside" implying "Let's go out").
ReCo (Representative $\to$ Commissive): A factual statement used to make a promise or commitment.
ReEx (Representative $\to$ Expressive): A factual statement used to express emotion or praise.

To test context dependency, the dataset pairs 80 items for each type into 40 pairs. In each pair, the same utterance is presented in two different scenarios: one where it functions as a direct speech act and another where it functions as an indirect speech act.

Models Evaluated

The study evaluated 12 LLMs, divided into two categories:

Proprietary Models (8): GPT-3.5, GPT-4, Claude3-Haiku, Claude3-Sonnet, Claude3-Opus, Mistral-small, Mistral-medium, and Mistral-large.
Open-Source Models (4): Llama3-8B, Qwen1.5-14B, Solar-10.7B, and T3Q-ko-solar (a fine-tuned Korean version of Solar).

Experimental Setup

Two experimental setups were employed to assess performance:

Multiple-Choice Question (MCQ): Models were presented with a context and utterance and asked to select the correct speech act intention from four options (one correct, one "opposite" type, and two random types). Performance was measured by the agreement rate (accuracy) across three trials per item.
Open-Ended Question (OEQ): Models were asked to describe the intention of the utterance. Responses were qualitatively scored by human evaluators on a 1–5 scale based on the accuracy of the inferred intent and literal meaning, then rescaled to 0–100.

Human Baseline

To establish a Gold Standard, 52 human participants (aged 20–30, diverse academic backgrounds, excluding linguistics majors) completed the same MCQ test. This data serves as the benchmark for comparing LLM performance against human pragmatic competence.

Key Contributions

Development of a Speech Act Comprehension Test Set: A specialized, publicly available dataset designed to evaluate LLMs' ability to interpret three categories of indirect speech acts within contextual frameworks in Korean.
Collection of Human Reference Data: The establishment of a human performance baseline (Gold Standard) to quantify the gap between current LLMs and human-level pragmatic understanding.
Systematic Evaluation and Analysis: A comprehensive assessment of 12 LLMs using both MCQ and OEQ formats, including a detailed analysis of performance across different speech act types (ReDi, ReCo, ReEx) and a comparison with conversational implicature understanding.

Results

Overall Performance

Human Superiority: Human participants achieved an average MCQ score of 77.64, significantly outperforming all evaluated LLMs.
Model Rankings: Claude3-Opus emerged as the top-performing model with an MCQ score of 71.94 and an OEQ score of 90.37. It was the only model to show balanced accuracy for direct and indirect speech acts, with a score difference of only 5.55% (69.17 for direct vs. 74.72 for indirect).
Proprietary vs. Open-Source: Proprietary models generally outperformed open-source models. Among open-source models, Solar-10.7B performed best (65.0 in MCQ), while T3Q-ko-solar (fine-tuned) actually showed a performance drop compared to the base Solar model in context-based inference.
Direct vs. Indirect Gap: Most LLMs (except Claude3-Opus) demonstrated significantly lower performance on indirect speech acts compared to direct ones. For instance, GPT-3.5 scored 63.61 on direct acts but only 36.67 on indirect acts.

Analysis by Speech Act Type

ReEx (Expressive): Both humans and LLMs achieved the highest scores on this type, likely due to the prevalence of idiomatic expressions in the dataset which models have learned well.
ReDi (Directive): This was the most challenging category for all models. Declarative sentences stating facts (e.g., "It's sold out") were frequently misinterpreted as information provision rather than indirect requests/suggestions.
ReCo (Commissive): Performance was intermediate, falling between ReDi and ReEx.

Qualitative Findings (OEQ)

Claude3-Opus demonstrated sophisticated reasoning, often providing detailed explanations and list formats to justify its interpretations.
Llama3 exhibited significant misalignment, frequently generating responses in English despite Korean prompts and producing unreadable tokens.
Solar showed signs of overfitting to MCQ benchmarks, often generating its own multiple-choice options (A, B, C) even in open-ended tasks, a behavior not typical of human problem-solving.

Correlation with Implicature

The study found a moderate correlation (0.67) between speech act understanding and conversational implicature understanding. However, the correlation was higher (0.73) specifically for indirect speech acts, suggesting that the mechanisms for interpreting indirect intentions and conversational implicatures are closely related, particularly regarding the Gricean maxim of relation.

Significance and Claims

The authors claim that this study provides a necessary pragmatic evaluation of LLMs, moving beyond knowledge-based benchmarks to assess the "highest level of language ability": pragmatic competence.

Current Limitations: The study concludes that while LLMs are advancing, no current model has reached human-level performance in understanding indirect speech acts. Most models struggle to distinguish between literal and implied intentions, particularly when the utterance is a factual statement used to make a request.
Context Sensitivity: The results highlight that models trained on large datasets excel at literal meaning but often fail to infer deeper intentions when the literal meaning is sufficient but contextually inappropriate.
Future Directions: The authors emphasize the need for further research to improve LLMs' understanding of indirect speech acts for more natural human-computer interaction. They propose future work involving simulated multi-turn conversations and the expansion of these datasets to other languages (English, Chinese, German) to validate robustness across linguistic contexts.

The paper positions itself as a foundational step in bridging the gap between statistical language modeling and true pragmatic understanding, arguing that rigorous evaluation of speech acts is essential for the next generation of conversational AI.