Medal Matters: Probing LLMs' Failure Cases Through Olympic Rankings

Technical Summary: Medal Matters: Probing LLMs' Failure Cases Through Olympic Rankings

Problem Statement
Despite the remarkable success of Large Language Models (LLMs) in Natural Language Processing, their internal knowledge structures and how they organize information remain poorly understood. A critical gap exists in understanding whether LLMs align with human reasoning patterns, particularly in connecting factual data to derived insights. While models often excel at retrieving specific facts, it is unclear if they can effectively integrate related knowledge to perform logical inferences, such as deriving rankings from medal counts. Furthermore, the robustness of these models against simple user expressions of doubt (e.g., "Really?") remains underexplored, raising concerns about their reliability in maintaining accurate responses when challenged without evidence.

Methodology
The study employs a structured analytical framework using historical Olympic medal data (1964–2022) to evaluate LLMs on two distinct tasks:

1. Medal QA: Retrieving exact medal counts for specific teams in specific Olympic Games (e.g., "How many medals did China win in the 2020 Tokyo Olympics?").
2. Team QA: Identifying the country that achieved a specific ranking in a given Olympic Games (e.g., "Which country ranked 3rd in the 2022 Beijing Winter Olympics?").

The dataset comprises medal results for 650 teams across 34 Olympic Games. The authors excluded the 2024 Paris Olympics (too recent for training data) and the 1960 Games (used as few-shot examples for formatting only). The evaluation involved 12 state-of-the-art (SOTA) models, including proprietary models (GPT-4o, GPT-4-turbo, Claude-3.5-Sonnet, Gemini-1.5-Pro) and open-source models (LLaMA-3.1, Qwen-2, Gemma-2).

Experiments were conducted in a "Closed-book" setup, where no external medal tables were provided as input. To assess robustness, a "Doubt Robustness" test was introduced: after the model's initial response, a prompt expressing doubt ("Really?") was appended, asking the model to reconsider its answer. Performance was measured by initial accuracy and final accuracy after the doubt prompt. A "Doubt Matrix" was utilized to categorize response changes into four cases: correct-to-correct, correct-to-incorrect, incorrect-to-incorrect, and incorrect-to-correct.

Key Results

1. Performance Disparity: A significant gap was observed between the two tasks. SOTA models demonstrated high accuracy in the Medal QA task (retrieving factual counts). However, performance dropped drastically in the Team QA task (identifying rankings), with no model exceeding 40% accuracy. The best performer, GPT-4o, achieved only 39.8% initial accuracy. This suggests that while LLMs can recall isolated facts, they struggle to organize and link related information (medal counts to rankings) in a structured manner similar to human reasoning.
2. Doubt Vulnerability: The study found that models are vulnerable to simple user doubt. In many instances, models altered their initially correct responses to incorrect ones after receiving a "Really?" prompt, leading to overall performance degradation. The "Doubt Matrix" revealed that at least 4.7% of total responses changed after doubt feedback, with a notable trend of correct answers being converted to incorrect ones.
3. Model Variance: While newer models (e.g., GPT-4o, Claude-3.5-Sonnet) exhibited slightly higher "doubt robustness" (maintaining correct answers more often), the general trend of performance decline upon user challenge persisted across the board.

Key Contributions

• Empirical Evidence of Structural Differences: The paper provides empirical evidence that LLM internal knowledge structures differ fundamentally from human reasoning, specifically regarding the integration of related facts to derive rankings.
• Introduction of "Doubt Robustness": The authors define and quantify "doubt robustness" as a critical evaluation metric, highlighting the tendency of LLMs to lose confidence in accurate answers when faced with unsupported user skepticism.
• Resource Release: To facilitate further research, the authors publicly released their code, dataset, and model outputs.

Significance and Claims
The paper claims that these findings shed light on critical limitations in the internal knowledge organization and robustness of LLMs. The inability to link related information suggests a fundamental limitation of the next-token prediction approach used in training. The vulnerability to doubt underscores the need for models that can maintain confidence in accurate answers without external validation.

The authors conclude that while LLMs are adept at retrieving specific factual information, they require further development to better structure internal knowledge and handle interconnected queries. They suggest that future work, such as incorporating graph-based approaches during pretraining, could help improve the organization and connection of information. The study emphasizes that enhancing the reliability and reasoning capabilities of LLMs is essential for building systems that users can trust in real-world scenarios.

Limitations
The authors acknowledge that their findings do not imply LLMs inherently lack the capacity to infer rankings, as advanced prompting strategies (e.g., Chain-of-Thought) might improve performance. The study focuses specifically on knowledge organization acquired during pretraining rather than reasoning with explicitly provided information. Additionally, the "doubt" mechanism used was a simple expression ("Really?"), which may not fully capture the nuances of complex real-world skepticism.