Ingredient-Level Food Image Segmentation for Nutrition Awareness

Technical Summary: Ingredient-Level Food Image Segmentation for Nutrition Awareness

Problem Statement
Current food recognition systems predominantly treat images as classification tasks, assigning a single dish label (e.g., "salad") to an entire image. This approach obscures the internal visual structure of meals containing multiple ingredients. For nutrition awareness, a single label fails to distinguish the spatial distribution of components such as rice, vegetables, meat, or sauces. While detailed calorie and macronutrient tracking is often too time-consuming for daily use, visual plate-based guidance (similar to the Healthy Eating Plate) offers a practical alternative by using approximate visual proportions. However, existing computer vision approaches often lack the granularity to separate visible ingredients at the pixel level. This study addresses the challenge of ingredient-level semantic segmentation on the FoodSeg103 dataset, aiming to predict an ingredient class for every pixel to generate a visible composition summary.

Methodology
The research focuses on fine-tuning SegFormer variants for the FoodSeg103 dataset, which contains 4,983 training images and 2,135 held-out test images with 104 class IDs (including background).

• Models: Two variants were evaluated under a strictly controlled setup to isolate the effect of model capacity:
  • SegFormer-B0 (MiT-B0): A smaller baseline model.
  • SegFormer-B1 (MiT-B1): A larger final model.
  • Both utilize ImageNet-pretrained Mix Transformer (MiT) backbones with newly initialized 104-class output layers.
• Training Setup: Both models shared identical hyperparameters and data splits to ensure a fair comparison. Key settings included an input size of 512×512, a batch size of 8, AdamW optimizer with a learning rate of 6e-5, and 40 epochs.
• Preprocessing: Images and masks were resized to 512×512. Bilinear interpolation was used for RGB images, while nearest-neighbor interpolation was applied to segmentation masks to preserve discrete class IDs. Data augmentation included random scaling, cropping, horizontal flipping, and mild color jitter.
• Evaluation: Models were selected based on the highest validation mean Intersection over Union (IoU). Final evaluation was performed on the held-out test set at the original mask resolution. Metrics included pixel accuracy, mean IoU, mean IoU excluding background, and mean class accuracy.
• Visible-Area Module: A post-processing step converts predicted masks into a "visible ingredient-area summary." It calculates the percentage of non-background pixels assigned to each ingredient class ($P_c = \frac{N_c}{N_{food}} \times 100$), providing a 2D composition overview.

Key Contributions

1. Reproducible Pipeline: Established a fine-tuning pipeline for ingredient-level segmentation on FoodSeg103 using SegFormer.
2. Controlled Comparison: Provided a direct comparison between SegFormer-B0 and B1, demonstrating the impact of increased model capacity while keeping data splits, training schedules, and evaluation code constant.
3. Comprehensive Evaluation: Reported held-out test metrics at original mask resolution, including detailed per-class IoU analysis.
4. Interpretable Output: Developed a module that translates dense segmentation masks into a readable 2D composition summary, offering a visual alternative to manual ingredient tracking.

Results
On the 2,135-image held-out test set, SegFormer-B1 outperformed the baseline B0 across all metrics:

• Pixel Accuracy: Increased from 0.7709 (B0) to 0.7929 (B1).
• Mean IoU: Improved from 0.2521 (B0) to 0.3204 (B1), an absolute gain of +0.0683.
• Mean Class Accuracy: Rose from 0.3590 to 0.4378.

Per-Class Analysis:

• High Performance: The model performed best on visually distinctive or large ingredients, achieving high IoUs for broccoli (0.8339), corn (0.8279), and green beans (0.7829).
• Low Performance: The model struggled with rare, small, or ambiguous classes, recording an IoU of 0.0 for items such as egg tarts, pudding, tea, and hamburgers. This aligns with known challenges in food segmentation, including class imbalance, overlap, and intra-class variation.
• Qualitative Findings: While B1 produced masks closer to ground truth, both models struggled with mixed dishes, thin regions, and unclear boundaries.

Significance and Limitations
The paper positions this work as a proof of concept for ingredient segmentation and visible composition analysis, rather than a complete nutrition estimation system.

• Significance: The primary value lies in interpretability. The system converts complex pixel-level predictions into a simple visual summary of meal composition (e.g., "61.8% carrot, 20.0% rice"). This serves as a "first-pass" nutrition awareness cue, potentially reducing the need for constant manual weighing or logging, similar to plate-based guidance.
• Modesty of Claims: The authors explicitly state that the system is not a direct estimator of calories, macronutrients, food mass, volume, density, or true portion size. A single RGB image lacks depth and density information.
• Limitations:
  • The study compares only two SegFormer variants and does not benchmark against other architectures like FCN, DeepLabV3+, U-Net, or BEiT.
  • Results are not directly comparable to published FoodSeg103 benchmark numbers due to the specific experimental setup.
  • Performance remains weak on rare and small classes.
  • The visible-area output is limited to 2D image area, which does not equate to actual nutritional mass.

In conclusion, the study demonstrates that SegFormer can be successfully fine-tuned for ingredient-level segmentation, with the larger B1 variant offering measurable improvements. While the system does not yet solve the complexities of full dietary estimation, it provides a foundational tool for visualizing meal composition to support nutrition awareness.