Technical Summary: OmniFood-Bench

Problem Statement

The integration of Large Vision-Language Models (VLMs) into personalized healthcare and dietary management faces a critical barrier: "Systemic Information Asymmetry." While autonomous agents can effectively recognize food categories, they struggle with the complex "Visual-to-Physical Inference" required for real-world dietary management. Specifically, current models fail to bridge the gap between visual appearance and intrinsic nutritional properties. They cannot reliably deduce invisible cooking methods (e.g., deep-fried vs. steamed), estimate physical mass from 2D pixels, or synthesize this data into safety-critical medical advice. Existing benchmarks (e.g., Food-101, VireoFood-172) focus on coarse-grained classification or recipe retrieval, treating food as semantic labels rather than physical objects with mass and chemical properties. Consequently, there is a lack of rigorous evaluation for the end-to-end reasoning chain necessary to prevent "Safety Hallucinations"—benign advice given to high-risk patients (e.g., diabetics) that could lead to adverse health outcomes.

Methodology

To address this gap, the authors introduce OmniFood-Bench, a comprehensive benchmark constructed from a curated subset of the MM-Food-100K dataset. The framework evaluates VLMs across three progressive capability layers designed to mimic the cognitive process of a human dietitian:

1. Data Construction: The benchmark comprises 1,208 high-quality samples across four modalities: Homemade Food, Restaurant Food, Packaged Food, and Raw Ingredients. Labels are defined based on authoritative health standards, categorizing intake levels (Normal, Controlled, Avoid) for specific clinical conditions (e.g., diabetes, hyperlipidemia, kidney disease).
2. Hierarchical Task Definitions:
   • Task I: Basic Perception: Evaluates semantic understanding through Cooking Method classification and Ingredient List generation (measured by Ingredient Match Rate).
   • Task II: Quantitative Reasoning: Probes the "Physical World Model" by requiring Portion Size estimation (in grams) and Nutritional Profile estimation (macros). Performance is measured using Mean Absolute Percentage Error (MAPE).
   • Task III: Safety-Critical Advisory: Acts as a clinical dietitian, requiring the model to output a risk-aware decision (Normal, Controlled, or Avoid) based on a specific user health profile.
3. Evaluation Protocol: Six state-of-the-art VLMs were evaluated in a zero-shot setting: three proprietary models (gpt-5.1, gemini-3-flash, claude-sonnet-4) and three open-weights models (qwen3-vl-8B, InternVL3 5-8B, Llama-3.2-11B-Vision).

Key Results

Extensive experiments reveal a stark "Semantic-Physical Gap" across all evaluated models:

• Perception vs. Quantification: While models achieve near-human accuracy in naming dishes and identifying ingredients (e.g., gpt-5.1 achieves ~87% accuracy on raw vegetables), their ability to estimate physical mass is catastrophic. MAPE for portion size estimation frequently exceeds 50%, reaching as high as 185% for raw vegetables and 314% for packaged food nutritional profiles.
• Dense Quantitative Failure: Accuracy collapses as the "Portion Size Variety Level" (number of distinct ingredients) increases. Current attention mechanisms struggle to perform multi-object disentanglement and mass regression simultaneously in cluttered scenes.
• Safety Hallucinations: The most critical failure occurs in the advisory stage. Models frequently generate "Safety Hallucinations," offering benign advice for high-risk foods. For instance, a model might correctly identify a glazed dish but fail to account for hidden sugars, recommending it as "Moderate Intake" for a diabetic user when it should be "Avoid."
• Low Advisory Accuracy: The best-performing models achieve only ~46% accuracy in the "Kidney Disease" category, barely surpassing random guessing in a 3-class classification task. There is a significant disconnect between accurate nutrient profiling and correct medical logic application.

Key Contributions

1. Unified Benchmark: The establishment of the first benchmark linking visual perception directly to medical safety outcomes, moving beyond simple accuracy metrics to "Risk-Aware" evaluation.
2. Comprehensive Analysis: A rigorous stress-test of six leading models, demonstrating that strong general capabilities do not automatically translate to safe dietary reasoning.
3. Failure Mode Diagnosis: The identification of the "Visual-to-Mass" estimation step as the primary bottleneck, which propagates errors downstream to health advice, and the quantification of "Safety Hallucinations" in biomedical contexts.

Significance

OmniFood-Bench establishes a rigorous standard for the trustworthiness of autonomous agents deployed in public health. The paper argues that "Safety Alignment" is not merely a quality metric but a fundamental requirement for deployment in high-stakes domains. By exposing the current inability of VLMs to perform physically grounded reasoning and adhere to clinical guidelines, this work highlights the necessity for future research to focus on Neuro-Symbolic Integration. The authors posit that combining the perceptual strengths of VLMs with structured nutritional knowledge bases and Retrieval-Augmented Generation (RAG) is essential to build agents capable of providing safe, accurate, and physically grounded dietary advice.