💻 computer science

DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding

This paper introduces DishSeg24k, a large-scale benchmark for food segmentation featuring 24,096 images and 278 fine-grained categories, alongside FEAST, a novel transformer model that leverages reinforcement learning-guided Mixture-of-Experts and stochastic expert decoding to achieve state-of-the-art performance in complex, long-tail dining scenarios.

Original authors: Yilin Wang, Haochen Shi, Guanyu Chen, Weiqing Min, Jinkai Zheng, Chenggang Yan, Shuqiang Jiang

Published 2026-08-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yilin Wang, Haochen Shi, Guanyu Chen, Weiqing Min, Jinkai Zheng, Chenggang Yan, Shuqiang Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world through its eyes. This field, called computer vision, is like giving a machine the ability to see, but with a twist: instead of just recognizing that "there is a cat," the robot needs to draw a perfect outline around every single part of that cat, separating its ear from its tail, and its tail from the rug it's sitting on. This is called image segmentation. It's the superpower behind self-driving cars knowing where the road ends and the sidewalk begins, or medical scanners spotting a tiny tumor. But while robots are getting pretty good at spotting cars and cats, they still struggle with a messy, delicious problem: food.

Why is food so hard for a robot? Because a plate of dinner isn't like a car; it's a chaotic jigsaw puzzle. Imagine a tray with rice, stir-fried beef, and tomatoes all squished together. The rice touches the beef, the beef touches the tomatoes, and they all share the same plate. To a robot, it's just a giant blob of brown and red. If the robot can't tell where the beef ends and the rice begins, it can't tell you how much protein you ate or how much the meal costs. This is the challenge of food segmentation: teaching a computer to untangle a messy dinner plate, figure out exactly which piece belongs to which dish, and draw a perfect line around it, even when the dishes are piled high and look suspiciously similar.

This is where a new study comes in, tackling the messiest dining tables in the world. The researchers, led by Yilin Wang and colleagues, realized that existing computer programs were failing because they were trained on neat, perfect photos of single dishes sitting alone on a white table. Real life, however, is a crowded buffet. To fix this, they built a massive new training library called DishSeg24k. This isn't just a few pictures; it's a collection of 24,096 images containing 112,281 separate food items and 278 different types of dishes, all captured in real, messy restaurants and canteens. They even included a long list of rare, regional specialties to make sure the robot doesn't just learn to recognize the most common foods like rice and noodles.

But having a great library isn't enough; the robot needs a new way of thinking. The authors proposed a new method called FEAST (Food Expert-Adaptive Segmentation Transformers). Think of the old way of teaching a robot as a rigid, step-by-step instruction manual: "Step 1: Look at the red spot. Step 2: Assume it's a tomato." If the robot makes a mistake in Step 1, it's stuck. FEAST, however, treats the process like a game of "20 Questions" played by a team of experts. Instead of following a single, straight path, the robot asks itself, "What if this red spot is actually a tomato sauce, or maybe a piece of beef?" It explores different possibilities, like a detective trying out different theories.

To make this work, the researchers gave the robot a "team of experts" inside its brain. Imagine a group of specialists: one is great at spotting liquid sauces, another is an expert on crunchy textures, and a third knows how to handle overlapping plates. When the robot sees a confusing pile of food, it doesn't just pick one expert; it lets a smart manager decide which experts to listen to for that specific moment. This is called a Mixture-of-Experts (MoE) system. To make sure this team stays balanced and doesn't all just agree on the most common food (like rice), the researchers used a technique inspired by Reinforcement Learning. It's like a coach giving the team a high-five (a reward) when they correctly separate a tricky boundary and a gentle "try again" when they get confused, helping them learn from their mistakes in real-time.

The results are impressive. When tested on their new, messy dataset, this new "team of experts" approach beat the previous best methods by a significant margin. It improved the accuracy of separating food items by 3.21% in one key measure, 3.68% in another, and 4.00% in a third. The researchers also tested their method on an older, smaller dataset called FoodSeg103, and it still performed better than the competition there, suggesting this new way of thinking could work for other types of messy images too.

In short, this paper doesn't just give us a bigger photo album of food; it gives robots a smarter, more flexible way to look at a messy dinner. By combining a massive, realistic dataset with a decision-making system that can explore different possibilities and learn from a team of specialized experts, the authors have shown a promising path toward computers that can truly understand the complex, delicious chaos of a real-world meal.

Technical Summary: DishSeg24k and FEAST

Problem Statement

Food segmentation is critical for intelligent catering, dietary assessment, and food recommendation systems. However, existing benchmarks and methods fail to address the unique complexities of real-world dining scenes. The authors identify three primary challenges that current approaches struggle with:

Dense Inter-dish Overlap: Real meals often involve multiple dishes served in shared containers or compartment trays, creating spatial overlaps and adjacencies that make delineating individual masks difficult.
Fine-grained Class Similarity: Different dish classes often share common ingredients (e.g., tomato in both "scrambled eggs with tomato" and "stir-fried beef with tomato"), leading to locally indistinguishable visual features even when boundaries are clear.
Extreme Long-tail Distributions: Natural consumption frequencies follow a severe long-tail pattern where common staples dominate training data while rare regional specialties have few instances, biasing models toward frequent classes.

Existing methods, including CNN-based architectures and query-based Transformers (e.g., Mask2Former), rely on deterministic decoding. This suppresses hypothesis diversity and limits robustness in ambiguous scenes. Furthermore, standard Mixture-of-Experts (MoE) approaches often suffer from "expert collapse" under long-tail distributions, where a few experts dominate while others are underutilized.

Methodology: FEAST

To address these challenges, the authors propose FEAST (Food Expert-Adaptive Segmentation Transformers), a framework that reformulates query-based decoding as a stochastic sequential decision process guided by Reinforcement Learning (RL).

1. MDP Formulation for Query Refinement

FEAST models the progressive update of object queries across $K$ decoder layers as a finite-horizon Markov Decision Process (MDP):

State ( $S$ ): The set of object queries at a given decoder layer.
Action ( $A$ ): A continuous semantic shift in the query embedding space.
Transition: A deterministic residual update where the new query is the sum of the current query and the action ( $q_{k+1} = q_k + a_k$ ).
Objective: To learn a stochastic policy that allows queries to explore multiple plausible semantic hypotheses before converging, rather than following a single deterministic trajectory.

2. Stochastic MoE Policy Network

The standard Feed-Forward Network (FFN) in the decoder is replaced with an MoE module to handle heterogeneous dish textures:

Architecture: Consists of one shared expert (capturing task-agnostic primitives) and $E_r$ routed experts (specializing in category-specific patterns).
Decoupled Routing: A router assigns queries to experts using a stop-gradient operator to prevent routing gradients from biasing the query features toward router-specific rather than task-relevant features.
Stochastic Sampling: The policy network outputs a Gaussian distribution over actions, allowing for exploration during training. At inference, the mean is used.

3. Decoupled Dual-Critic Optimization

To stabilize training under long-tail distributions and prevent expert collapse, FEAST introduces a dual-reward structure with two independent critics:

Task Reward ( $r_{task}$ ): Evaluates incremental improvements in mask quality (Dice) and classification confidence. It is reweighted by the inverse square root of class frequency to address long-tail bias.
Structure Reward ( $r_{struct}$ ): Evaluates the router's load-balancing efficiency (entropy and uniformity of expert activation) independent of mask quality. This acts as a tiebreaker to encourage diverse expert usage without interfering with segmentation quality.
Optimization: The query policy is optimized via policy gradient with KL regularization against a frozen reference policy (pretrained Mask2Former), while the routing policy is optimized via REINFORCE using the structure advantage.

Key Contributions

DishSeg24k Dataset: A large-scale benchmark comprising 24,096 images, 112,281 instances, and 278 fine-grained categories. It is constructed from real-world dining scenes (including Chinese and Western styles) and features high instance density, severe inter-dish overlap, and a long-tail distribution. The dataset includes rigorous multi-stage quality control and hierarchical labeling.
FEAST Framework: A novel decoder architecture that treats query updates as an RL-optimized MDP. It integrates an MoE decoder with a decoupled dual-critic scheme to simultaneously improve uncertainty-aware mask refinement and structure-aware expert routing.
Comprehensive Evaluation: Extensive experiments on DishSeg24k comparing FEAST against 13 representative methods (CNN-based, per-pixel Transformers, and query-based models), along with validation on the FoodSeg103 dataset to demonstrate generalizability.

Experimental Results

Performance on DishSeg24k

FEAST achieves state-of-the-art performance on the proposed benchmark:

ResNet-50 Backbone: FEAST outperforms the strongest baseline (Mask2Former) by 3.21% mIoU, 3.68% mDice, and 4.00% mAcc.
Swin-B Backbone: FEAST achieves 55.07% mIoU, 74.26% mDice, and 63.36% mAcc.
Ablation Studies:
- The MoE decoder provided the largest individual gain (+3.03% mIoU).
- RL guidance was essential to convert stochastic perturbation into controlled exploration, yielding the final performance boost.
- The dual-critic optimization was shown to be necessary; partial RL configurations were unstable, while the joint configuration improved performance over the MoE-only baseline.
- Optimal performance was achieved with 4 experts; increasing to 8 experts reduced performance due to signal fragmentation.

Generalization on FoodSeg103

FEAST was validated on the FoodSeg103 dataset (ingredient-level annotations):

With ResNet-50, it improved mIoU by 1.47% over Mask2Former.
With Swin-B, it improved mIoU by 2.92% over FDSNet.
These results confirm that the RL-guided MoE decoding strategy generalizes beyond the specific challenges of DishSeg24k.

Significance

The paper claims that handling semantic entanglement in food segmentation requires not only stronger representations but also dynamic inference. By moving away from deterministic query refinement toward a stochastic, RL-guided process, FEAST effectively manages the ambiguity caused by inter-dish overlap and fine-grained class similarity. The introduction of DishSeg24k provides a necessary testbed for future research, as existing benchmarks fail to capture the compositional complexity of authentic dining scenes. The work suggests that decoupling task-oriented refinement from structure-aware routing is a viable strategy for training robust models on long-tail, highly overlapping data distributions.