💻 computer science

OmniFood8K: Single-Image Nutrition Estimation via Hierarchical Frequency-Aligned Fusion

This paper introduces OmniFood8K, a comprehensive multimodal dataset addressing the lack of Chinese cuisine data, alongside a large-scale synthetic dataset and an end-to-end framework that leverages depth estimation and frequency-aligned fusion to achieve accurate single-image nutrition estimation.

Original authors: Dongjian Yu, Weiqing Min, Qian Jiang, Xing Lin, Xin Jin, Shuqiang Jiang

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dongjian Yu, Weiqing Min, Qian Jiang, Xing Lin, Xin Jin, Shuqiang Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're sitting at a restaurant, looking at a delicious plate of food. You want to know: How many calories is this? How much protein? Is it healthy?

Usually, you'd have to guess, look up a recipe, or ask a nutritionist. But what if your phone camera could just snap a photo and tell you the exact nutritional breakdown instantly? That's the dream this paper, "OmniFood8K," is trying to make real.

Here is the story of how they did it, explained without the jargon.

1. The Problem: The "Western Bias" and the "Depth Camera" Trap

The researchers found two big problems with existing food apps:

The Menu is Wrong: Most food databases are like a menu that only lists burgers and pizza. They are terrible at understanding complex Chinese dishes (like a stir-fry with ten different ingredients). If you take a photo of a traditional Chinese meal, old apps get confused.
The Camera is Too Picky: The best apps right now need a special "3D depth camera" (like the ones on some iPhones) to measure how tall the food is. But most of us just have regular cameras. If you don't have the special camera, these apps don't work.

2. The Solution: A Massive New Library (OmniFood8K)

To fix the "wrong menu" problem, the team built a new, massive library called OmniFood8K.

What's inside? They didn't just take random photos. They went into real kitchens, weighed every single ingredient (like 100g of pork, 20g of garlic), filmed the cooking process, and took photos of the final dish from six different angles.
The Result: They have 8,000+ meals with perfect nutritional math attached to them. It's like having a super-accurate nutritionist who watched every step of the cooking.
The "Fake" Library (NutritionSynth-115K): To make the computer even smarter, they created a "fake" library of 115,000 images. They took pieces of real food and mixed them together digitally to create new meals. This taught the AI to recognize ingredients even when they are mixed in weird ways.

3. The Magic Trick: Seeing in 3D with a 2D Camera

The biggest challenge was: How do you measure the volume of a pile of rice from a flat, 2D photo?

The researchers built a three-step "magic trick" (their AI framework) to solve this:

Step A: The "Mind's Eye" (Depth Estimation)

Since the camera is flat, the AI first has to guess how deep the food is. It uses a "Mind's Eye" model to imagine a 3D map of the plate.

The Problem: The guess is usually a bit wobbly. Maybe it thinks the rice is a mountain when it's actually a hill.
The Fix (SSRA): They added a "Tuning Knob" (Scale-Shift Residual Adapter). This tool acts like a photo editor that says, "Whoa, that looks too tall, let's shrink it a bit," or "That edge looks blurry, let's sharpen it." It makes the 3D guess look real and consistent.

Step B: The "Frequency Mixer" (FAFM)

Now the AI has two things: the original photo (RGB) and the guessed 3D map (Depth). It needs to combine them.

The Analogy: Imagine listening to a song. You have the bass (the deep, overall structure) and the treble (the crisp, high details).
The Fix: The AI uses a "Frequency Mixer" (Frequency-Aligned Fusion Module). It separates the "bass" (the overall shape of the plate) from the "treble" (the texture of the broccoli). It mixes the best parts of the photo and the 3D guess together so they don't fight each other. This helps the AI understand the food's shape much better.

Step C: The "Spotlight" (MPH)

Finally, the AI has to calculate the numbers.

The Problem: A plate has rice, meat, sauce, and a garnish. The AI shouldn't waste brainpower analyzing the empty space on the plate.
The Fix: They added a "Spotlight" (Mask-based Prediction Head). This tool automatically turns on a spotlight on the important ingredients (the meat and veggies) and turns off the lights on the boring parts. It focuses only on what matters to get the calorie count right.

4. The Result: Better Than the Pros

They tested this new system against all the other top methods.

The Verdict: Their method was the most accurate, even though it only used a regular phone camera.
Why? Because they taught it with a massive, diverse library of real Chinese food, and they gave it a clever way to "see" 3D depth from a flat picture.

Summary

Think of this paper as building a super-smart nutritionist that lives in your phone.

It learned by studying 8,000 real meals (OmniFood8K) and 115,000 fake meals (NutritionSynth-115K).
It uses a tuning knob to fix its 3D guesses.
It uses a mixer to combine the photo and the 3D guess perfectly.
It uses a spotlight to focus only on the food, ignoring the plate.

Now, you don't need a special camera or a nutritionist degree. You just snap a photo, and the AI tells you exactly what you're eating.

1. Problem Statement

Accurate estimation of food nutrition is critical for personalized diet management and preventing chronic diseases. However, existing approaches face two fundamental limitations:

Data Limitations: Most existing datasets (e.g., Nutrition5k, Food-101) are heavily biased toward Western cuisines. They lack sufficient coverage of complex Chinese dishes, which have diverse ingredients and non-standardized preparation methods, making accurate annotation and nutritional analysis difficult.
Algorithmic Limitations: State-of-the-art methods often rely on depth sensors (RGB-D) to estimate food volume and mass. In real-world daily scenarios, users typically capture food photos using standard RGB cameras without depth sensors, rendering these depth-dependent methods impractical for widespread adoption.

2. Methodology

The authors propose an end-to-end framework that predicts nutritional information (Mass, Calories, Protein, Fat, Carbohydrates) directly from a single RGB image. The pipeline consists of three core modules:

A. Depth Estimation and Refinement (SSRA)

Since RGB images lack explicit depth cues, the system first uses a pre-trained monocular depth estimation model to generate an initial depth map. To correct the inherent scale ambiguity and local structural errors of monocular depth, they introduce the Scale-Shift Residual Adapter (SSRA):

Global Alignment: Applies a learnable scaling factor ( $\alpha$ ) and shift parameter ( $\beta$ ) to align the overall depth distribution ( $d_{global} = \alpha \cdot d_{mono} + \beta$ ).
Local Refinement: Uses a shallow convolutional residual module to correct geometric distortions and enhance local structural details.
Output: Produces a refined depth map that ensures both global scale consistency and local structural preservation.

B. Frequency-Aligned Fusion (FAFM)

To effectively integrate the RGB image and the refined depth map, the authors propose the Frequency-Aligned Fusion Module (FAFM). This module operates in the frequency domain:

Decomposition: Uses 2D Fast Fourier Transform (FFT) to separate features into low-frequency (global structure) and high-frequency (local texture/details) components.
Fusion: Fuses the low-frequency and high-frequency components from both modalities via element-wise addition.
Alignment: Introduces an inter-modal alignment loss ( $L_{align}$ ) based on contrastive learning to enforce semantic consistency between RGB and depth features, ensuring they represent the same sample closely in the feature space.

C. Mask-based Prediction Head (MPH)

The final stage predicts nutritional values using a Mask-based Prediction Head (MPH):

Feature Integration: Takes multi-scale RGB features and semantic features as input.
Dynamic Selection: Employs a Cross-Attention mechanism and a Gated Fusion strategy. Crucially, it uses a dynamic channel mask to automatically select the most informative channels, emphasizing key ingredient regions relevant to the target nutrients.
Output: Aggregates features via global pooling and fully connected layers to output the final nutritional values.

D. Loss Function

The training utilizes a Dynamic Task-weighted Loss for nutrient estimation, which adjusts weights based on the prediction difficulty of each nutrient (using a KPI metric based on Percentage Mean Absolute Error). This is combined with the inter-modal alignment loss.

3. Key Contributions

A. OmniFood8K Dataset

A comprehensive multimodal dataset containing 8,036 food samples focused on Chinese cuisine. Unlike previous datasets, it covers the entire food preparation pipeline:

Data Types: Raw ingredient images with precise weights, structured recipe descriptions, cooking videos, and multi-view images of finished dishes.
Annotations: Detailed nutritional labels (Calories, Protein, Fat, Carbs, Mass) calculated based on real-world weight measurements and official nutritional databases, ensuring high accuracy.
Significance: It is the first dataset to preserve causal links from raw ingredients to final presentation, addressing the lack of Chinese food data.

B. NutritionSynth-115K

A large-scale synthetic dataset containing 115,000 images.

Construction: Created by segmenting, cropping, and recombining items from OmniFood8K while preserving precise nutritional labels.
Purpose: Provides compositional diversity to pre-train models, enhancing their generalization capabilities before fine-tuning on real-world data.

C. Novel Architecture

The proposed framework is the first to achieve high-accuracy nutrition estimation from a single RGB image without requiring depth sensors, utilizing hierarchical frequency-aligned fusion and dynamic channel selection.

4. Experimental Results

The method was evaluated on OmniFood8K and the standard Nutrition5k dataset, using Percentage Mean Absolute Error (PMAE) as the primary metric.

Performance on OmniFood8K: The proposed method achieved a mean PMAE of 8.6%, significantly outperforming strong baselines like ResNet-101 (10.6%) and Swin-Transformer (31.2%).
Performance on Nutrition5k: The method achieved a mean PMAE of 16.5%. Notably, it outperformed existing state-of-the-art methods that rely on RGB-D inputs (e.g., FBFPN at 17.3%, IMIR-Net at 17.4%), proving that the single-RGB approach with their specific fusion strategy is superior to depth-dependent methods.
Ablation Studies:
- Adding SSRA, FAFM, and MPH incrementally improved performance.
- Pre-training on NutritionSynth-115K significantly boosted performance on OmniFood8K, reducing the mean PMAE from 8.6% to 7.8%, demonstrating the value of the synthetic dataset for initialization.

5. Significance

Practical Applicability: By removing the dependency on depth sensors, this method enables accurate nutritional tracking using standard smartphone cameras, making it viable for daily consumer use.
Cultural Inclusivity: The introduction of OmniFood8K addresses the critical gap in Chinese cuisine data, enabling more accurate health management for a large global population.
Technical Advancement: The use of frequency-domain alignment and dynamic channel masking offers a new paradigm for multi-modal fusion in computer vision, specifically for tasks requiring precise geometric and semantic understanding from 2D images.