SommBench: Assessing Sommelier Expertise of Language Models

The paper introduces SommBench, a multilingual benchmark developed in collaboration with professional sommeliers to evaluate the sensory expertise of language models across wine theory, feature completion, and food-wine pairing tasks, revealing that while models excel at theoretical knowledge, they struggle with more complex sensory judgment challenges.

The Gist

Imagine you have a super-smart robot that has read almost every book, website, and article on the internet. It can write poetry, solve math problems, and chat in dozens of languages. But there's a big question: Does this robot actually understand the world, or is it just a really good parrot repeating what it's heard?

Specifically, can this robot act like a Sommelier—a wine expert who knows not just facts about wine, but also how to describe its taste, smell, and which foods it goes best with?

This paper introduces SommBench, a giant "final exam" designed to test if AI can truly be a wine expert. Here's the breakdown in simple terms:

🍷 The Exam: Three Different Tests

The researchers didn't just ask the AI one question. They gave it three distinct challenges, like a sommelier training program:

1. The Trivia Test (Wine Theory):

   • The Task: Answer multiple-choice questions about wine facts (e.g., "What grape is used in Chianti?").
   • The Twist: The questions are asked in 8 different languages (English, German, Spanish, etc.).
   • The Goal: To see if the AI knows the facts no matter what language you speak.
   • The Result: The AI aced this! The smartest models got about 97% right. It's like a student who memorized the entire encyclopedia.

2. The Fill-in-the-Blanks Test (Wine Features):

   • The Task: You give the AI a partial profile of a wine (e.g., "It's from France, it's red, and has high alcohol...") and ask it to guess the missing details (e.g., "What is the grape variety?").
   • The Twist: It has to do this in different languages and output the answer in a strict, structured format.
   • The Goal: To see if the AI can connect the dots and infer missing information logically.
   • The Result: This was harder. The best AI got about 65% right. It's like a student who knows the facts but sometimes guesses the wrong details when the clues are tricky.

3. The Dinner Party Test (Food & Wine Pairing):

   • The Task: You give the AI a recipe (e.g., "Spicy Thai Curry") and a bottle of wine. The AI must say: "Yes, this is a great match" or "No, this is a terrible match."
   • The Twist: This requires taste and judgment, not just facts. It's subjective.
   • The Goal: To see if the AI has "good taste" or if it's just guessing.
   • The Result: This was the biggest failure. The best AI only got about 39% of the "right" answers (using a special scoring method). Many models were worse than random guessing.

🤖 The Big Findings

1. The "Parrot" Problem (Language vs. Knowledge)
The smartest AI models (the "closed" ones, like the ones you pay for) were great at knowing wine facts in English. But when you switched to languages like Slovak or Finnish, some of the "open" models (free to use) suddenly forgot everything.

• Analogy: Imagine a student who is a genius in English class but suddenly acts like they've never heard of "France" when you ask the question in German. It turns out, for many AIs, their knowledge is tied to the language they learned it in, not the actual concept.

2. The "Yes-Man" Bias
When it came to pairing food and wine, the AI had a weird personality flaw: It was too nice.

• The Issue: If you asked, "Does this terrible wine go with this fancy steak?" the AI would often say "Yes!" just to be helpful.
• Analogy: It's like a waiter who is so afraid of offending you that they tell you a burnt steak is "crispy and delicious." The AI has a "positivity bias"—it prefers to agree and recommend things rather than say "No, that's a bad idea."

3. Facts vs. Feelings
The study proved that AI is great at facts (reciting data) but terrible at feelings (judging taste).

• Analogy: An AI can tell you the chemical formula of a strawberry, but it can't tell you if a strawberry tastes better with cream or with salt. It hasn't actually tasted anything; it's just reading descriptions of tasting.

🏁 The Verdict: Should You Trust an AI Sommelier?

For facts? Yes. If you want to know the history of a vineyard or the alcohol content of a wine, the best AI models are incredibly reliable.

For dinner advice? No. If you ask an AI, "What wine should I drink with my spicy fish tacos?" it might give you a recommendation that sounds good but tastes awful. It lacks the human "gut feeling" and sensory experience that makes a real sommelier valuable.

In short: SommBench shows us that while AI is getting smarter, it's still a bit like a bookish student who has never actually eaten a meal. It knows the menu, but it doesn't know the flavor.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) have demonstrated strong performance on general multilingual benchmarks, a critical gap remains in evaluating their culturally-grounded, expert-level knowledge across different languages. Specifically, it is unclear whether LLMs possess consistent competencies in specialized domains (like sommelier expertise) or if their performance degrades due to uneven training data distributions across languages.

Current benchmarks often focus on general cultural literacy or basic factual recall. There is a lack of evaluation for:

• Sensory Judgment: Can models emulate expert-level sensory judgments (taste, smell) learned purely from text?
• Cross-lingual Consistency: Does a model provide contradictory information about the same wine when queried in different languages?
• Subjective Expertise: Can models distinguish between harmonious and discordant food-wine pairings, a task requiring nuanced cultural and sensory reasoning?

2. Methodology: SommBench

The authors introduce SommBench, a novel, multilingual benchmark designed to evaluate LLMs in the domain of professional sommelier expertise. The benchmark was developed in close collaboration with a professional master sommelier and native speakers of eight languages: English, Slovak, Swedish, Finnish, German, Danish, Italian, and Spanish.

The benchmark consists of three complementary tasks:

A. Wine Theory Question Answering (WTQA)

• Goal: Test factual knowledge recall and cross-lingual consistency.
• Data: 1,024 multiple-choice questions (128 per language) derived from established sommelier exams (e.g., WSET).
• Process: Models answer questions covering oenological theory, regional classifications, and varietal properties. Distractors are crafted to reflect common misconceptions.
• Evaluation: Accuracy scores calculated per language and an overall multilingual average.

B. Wine Feature Completion (WFC)

• Goal: Assess the ability to infer missing structured attributes from partial data.
• Data: 1,000 wine entries with nine key attributes (type, sugar, alcohol, country, region, grapes, dryness, body, acidity).
• Process: A tiered masking strategy is used:
  • Single Mask: Tests factual recall.
  • Double Mask: Tests simple inference (e.g., inferring country from region).
  • Triple Mask: Tests complex compositional completion under high uncertainty.
• Evaluation: Exact match for categorical fields; Mean Absolute Percentage Error (MAPE) for numerical fields (alcohol, sugar).

C. Food-Wine Pairing (FWP)

• Goal: Evaluate culturally adaptive reasoning and subjective expert judgment.
• Data: 1,000 expert-validated pairings (positive and negative) involving specific recipes and wines. Note: This task is monolingual (English-only) due to the complexity of translating full recipe descriptions while preserving culinary nuance.
• Process: Models act as a binary classifier, determining if a specific wine-recipe combination is a "good" pairing (Yes/No).
• Evaluation: Matthews Correlation Coefficient (MCC) is used as the primary metric to provide a balanced assessment of both positive and negative predictions, avoiding the bias of F1 scores.

Scoring

A unified SommBench Score is calculated as the arithmetic mean of the performance across the three tasks (WTQA, WFC, FWP), with multilingual tasks averaged across all eight languages.

3. Key Contributions

1. Novel Benchmark: Introduction of SommBench, the first multilingual benchmark specifically targeting sommelier expertise, comprising 3,024 examples across three distinct task types.
2. Expert Curation: The dataset is not merely translated; it is curated by a master sommelier to ensure authentic, expert-level complexity and to create "distractors" and pairings that are not publicly available in pre-training corpora.
3. Cross-Lingual Analysis: The framework explicitly separates language skills from domain expertise, allowing for the detection of inconsistencies in how models handle the same factual or subjective knowledge across different languages.
4. Bias Discovery: The study identifies and quantifies a "positivity bias" in LLMs regarding food-wine pairings, where models tend to approve pairings regardless of compatibility.

4. Experimental Results

The authors evaluated 18 models, including leading closed-weight models (e.g., Gemini 2.5, GPT-5, Grok-4) and open-weight models (e.g., Qwen 3, Llama 3.1, GPT-OSS).

• Overall Performance: Even the best-performing model, Gemini 2.5-flash, achieved a total SommBench Score of only 0.65. Open-weight models peaked at 0.52 (Qwen 3:30b).
• Task Difficulty:
  • WTQA (Easiest): Top closed models achieved near-perfect accuracy (>95%), indicating strong assimilation of factual wine knowledge.
  • WFC (Moderate): Performance dropped significantly (peaking at ~65%), highlighting the difficulty of structured multilingual generation and inference.
  • FWP (Hardest): This was the primary differentiator. Top models scored an MCC of only 0.39, while several models scored near or below zero (random chance), indicating they cannot reliably emulate expert judgment.
• Closed vs. Open Weights: Closed-weight models consistently outperformed open-weight models.
• Cross-Lingual Consistency:
  • Closed Models: Maintained high stability across all 8 languages.
  • Open Models: Showed significant performance degradation in non-English languages. For example, Llama 3.1:8b dropped from 0.70 accuracy in English to 0.27 in Slovak for WTQA.
• Reasoning Impact: Increasing reasoning intensity (tested on GPT-OSS) improved factual recall (WTQA) but degraded performance on subjective tasks (FWP) and structured generation (WFC), suggesting "overthinking" harms complex judgment tasks.
• Positivity Bias: Many models exhibited a strong bias toward approving pairings. For instance, GPT-4o-mini approved 86% of pairings despite a 50/50 ground truth, resulting in an 82% false-positive rate on bad pairings.

5. Significance and Conclusion

• Benchmarking Expertise: SommBench proves that while LLMs have mastered factual retrieval in specialized domains, they struggle with subjective judgment and cross-lingual consistency in culturally situated tasks.
• Limitations of Current AI: The results suggest that deploying current frontier models as "virtual sommeliers" for meal recommendations is inadvisable, as they fail to distinguish good pairings from bad ones and exhibit inherent approval biases.
• Research Directions: The paper highlights the need for:
  • Better multilingual training to ensure cross-lingual consistency.
  • Moving beyond binary classification to generative pairing tasks.
  • Incorporating diverse expert annotations to handle the subjectivity of sensory evaluation.

In summary, SommBench serves as a rigorous stress test for LLMs, revealing that while they can recite wine theory, they have not yet mastered the nuanced, culturally dependent art of sommelier expertise.