💻 computer science

Multi-Scale Fruit Capsules: Dilated Convolutions and Dynamic Routing for In-the-Wild Explainable Fruit Recognition

This paper introduces FruitCapsNet, a multi-scale capsule network utilizing dilated convolutions and Bayesian-optimized hyperparameters to achieve state-of-the-art, explainable fruit recognition in diverse in-the-wild conditions by preserving spatial pose information and focusing on whole-fruit regions rather than edges.

Original authors: Subhankar Chattoraj, Sawon Pratiher, Samiran Das, Hubert Konik

Published 2026-08-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Subhankar Chattoraj, Sawon Pratiher, Samiran Das, Hubert Konik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the bustling world of modern agriculture, the journey of a fruit from tree to table is increasingly managed by machines. Farmers and packing facilities rely on automated systems to sort produce, check for ripeness, and identify defects with a speed and consistency that human hands cannot match. For these machines to work, they must be able to "see" fruit the way humans do, recognizing an apple whether it is hanging on a branch, sitting in a plastic bag, sliced on a plate, or bruised by a fall. This task is deceptively difficult because the same fruit can look radically different depending on its condition, the lighting, and what surrounds it. While computers have become incredibly good at spotting objects in photos, they often struggle when the object is not in a perfect, isolated pose. Traditional computer vision systems tend to focus on small, local details, like the curve of a skin or a specific color patch, and then piece those fragments together. This approach works well in controlled settings but often fails in the messy, unpredictable reality of a farm or a grocery store, where fruits overlap, shadows fall, and backgrounds are cluttered.

A team of researchers has developed a new way to teach computers to see fruit that mimics the human ability to understand the whole object rather than just its parts. They created a system called FruitCapsNet, designed specifically to handle the chaotic variety of real-world fruit photography. Instead of using the standard methods that have dominated the field for years, which often discard important information about where a part of an object is located, this new system keeps track of the position and orientation of every piece of the fruit. It does this by using a specialized type of digital filter that allows the computer to see a much wider area around each point of interest without needing to add more processing power. By combining this wide view with a method that checks how well different parts of the fruit agree with each other to form a whole, the system can recognize a fruit even when it is partially hidden or surrounded by other objects.

The researchers tested their system on four different collections of fruit images, ranging from clean, studio-style photos to a new, challenging set of over ten thousand images taken in the wild. This new set, which they named PD-19, contains images with multiple fruits in a single frame, uneven lighting, and fruits in various states like peeled, bagged, or sliced. When they compared their system against ten of the most powerful existing computer vision models, FruitCapsNet performed better on every single dataset. The difference was most dramatic on the difficult, real-world images, where the new system outperformed the next best competitor by nearly three percentage points. In the world of automated sorting, where millions of items are processed, even a small improvement in accuracy can prevent significant waste and financial loss. The researchers found that their system was not just guessing; it was actually looking at the entire fruit to make its decision, rather than focusing on the edges or background noise, which is a common mistake made by older systems.

To understand why this works, one must look at how the system processes an image. Traditional systems often break an image down into small tiles and decide if a feature exists in a tile, ignoring exactly where that feature sits within the tile. This is like recognizing a face only by the shape of the nose, without caring if the nose is on the left or right side of the face. The new system, however, uses a structure that preserves the relationship between parts. It uses a technique called dilated convolution, which acts like a lens that expands the view of the computer, allowing it to see the context around a fruit without losing the fine details. This means that when the system looks at a sliced orange, it understands that the segments belong to a single round object, even if the background is a busy kitchen counter. The system then uses a process called dynamic routing to vote on whether these parts fit together correctly to form a specific type of fruit. If the parts agree on the shape and pose of the whole fruit, the system becomes confident in its identification.

The team also spent considerable time fine-tuning the system's internal settings, not by guessing or trying every possible combination, but by using a smart mathematical search method that learns from each attempt. This allowed them to find the perfect balance for how the system weighs different types of errors and how it adjusts its learning over time. The result is a model that is much smaller and faster than the massive systems usually required for this level of accuracy, using only a fraction of the computational depth. When the researchers looked at the "heat maps" generated by the system to see what it was focusing on, they saw a clear difference. The older systems tended to highlight the edges of the fruit or the shadows around it, while the new system consistently highlighted the entire fruit, from its center to its skin. This suggests that the system is truly learning the concept of the fruit as a whole object, making it far more robust against the confusion of real-world environments.

The study confirms that by changing how a computer network processes visual information, it is possible to achieve a level of understanding that was previously out of reach for automated systems. The researchers demonstrated that their approach works across a wide variety of fruit types and conditions, from the 15 classes in one dataset to the 19 classes in their new wild dataset. While the system is not perfect and still faces challenges with extremely noisy data or the high cost of running complex calculations on small devices, the results show a clear path forward. The work suggests that the future of automated fruit recognition lies not in making systems bigger and deeper, but in making them smarter about how they piece together the visual world. By respecting the spatial relationships between parts and understanding the context of the whole, machines can finally learn to see fruit the way we do, ready to handle the messy, beautiful reality of the harvest.

Technical Summary: Multi-Scale Fruit Capsules for In-the-Wild Explainable Fruit Recognition

Problem Statement
Automated fruit classification in the wild (AFCW) faces significant challenges due to high intra-class variability and narrow inter-class separation. Fruits exhibit diverse appearances based on ripening stages, seasons, and provenance, and they are often photographed in various states: unpeeled, peeled, sliced, bagged, or sliced on a dish. Standard Convolutional Neural Networks (CNNs) struggle with this variability because their routing mechanisms (pooling) discard the precise pose and location of features, treating objects as "bags of patches." Consequently, CNNs lack the ability to represent the spatial relationships between parts and the whole object, which is critical for distinguishing, for example, a sliced fruit on a plate from the same fruit in a bag. While alternative sensing modalities (e.g., hyperspectral, thermal) exist, they are often cost-prohibitive or invasive, leaving color imaging as the primary economic option and placing the burden of robustness on the model architecture.

Methodology
The authors propose FruitCapsNet, a capsule network designed to address the limitations of standard CNNs through three core architectural and methodological innovations:

Dilated Fruit Capsules: The standard convolutional front-end of capsule networks is replaced with dilated convolutions. While standard convolutions have a receptive field limited by kernel size and depth, dilated convolutions expand the receptive field exponentially at a constant parameter cost. This allows each capsule to aggregate multi-scale context (spanning the fruit and its immediate background) before the dynamic routing process resolves part-whole spatial agreement. This pre-routed context is crucial for handling occlusion and multi-object scenes.
Dynamic Routing with Vector Encapsulation: Unlike CNNs that output scalar feature maps, FruitCapsNet utilizes vector-in/vector-out capsules. The length of the vector represents the probability of a class's presence, while the direction encodes instantiation parameters (pose, deformation, albedo). Dynamic routing replaces pooling, iteratively refining the coupling coefficients between child and parent capsules to ensure that only child capsules agreeing on a consistent pose for the whole object contribute significantly to the parent's activation.
Bayesian Hyper-parameter Optimization: The model employs Bayesian optimization with a Gaussian-process surrogate to select hyper-parameters, including the margin-loss constants ( $m^+, m^-, \lambda, \alpha$ ), dilation factors, and training schedules. This approach replaces manual grid search, optimizing the interaction of four loss constants alongside network depth, filter sizes, and learning rates.

The architecture consists of an encoder (three dilated convolutional layers, a PrimaryCaps layer, and a DigitCaps layer) and a decoder that reconstructs the input image to act as a regularizer against overfitting.

Key Contributions

Architectural Novelty: The paper introduces the first application of dilated convolutions within the front end of a capsule network. This design allows capsules to capture multi-scale context prior to routing, a gap not previously addressed in capsule literature.
PD-19 Benchmark: The authors introduce PD-19, a new in-the-wild dataset comprising 10,639 images across 19 classes. Unlike existing datasets (SMP, FruitsGB, Fruits-360) which often feature single fruits on homogeneous backgrounds, PD-19 includes multiple categories per image, inhomogeneous backgrounds, occlusion, and varied presentations (cut, plated, bagged).
Explainability via Grad-CAM: The study adapts Grad-CAM to capsule networks by propagating gradients from the DigitCaps layer back to the convolutional layer. This provides visual evidence that the model's decisions are attributed to whole-fruit regions rather than just object edges.
Optimization Strategy: The use of Bayesian optimization to tune the complex, non-differentiable loss landscape of capsule networks, specifically the margin-loss constants and dilation factors.

Experimental Results
The model was evaluated on three public datasets (SMP, FruitsGB, Fruits-360) and the new PD-19 dataset.

Performance: FruitCapsNet outperformed ten fine-tuned transfer-learning backbones (including ResNet, VGG, DenseNet, and MobileNet variants) across all datasets. Notably, it achieved these results with a network depth of only 7 layers compared to the 201 layers of the nearest competitor (DenseNet-201).
Difficulty Scaling: The performance margin scaled with scene difficulty. On the saturated public datasets, improvements were marginal (e.g., +0.32% on FruitsGB). However, on the challenging PD-19 dataset, FruitCapsNet exceeded the nearest competitor by +2.67%.
Ablation Studies:
- Dilation: Setting the dilation factor to $d=2$ accounted for 50–62% of the total improvement on public datasets and 70% on PD-19. Increasing to $d=3$ degraded performance due to gridding artifacts and excessive background noise integration.
- Bayesian Optimization: Provided an additive, dataset-agnostic boost of +0.29% to +0.69%, optimizing the learning schedule rather than spatial context.
Visual Analysis: Grad-CAM visualizations demonstrated that while DenseNet's saliency collapsed onto object edges, FruitCapsNet attributed decisions to the entire fruit region. This whole-object attribution remained robust under occlusion, clutter, and pose changes, whereas the baseline's heatmaps became ambiguous or drifted to the background.

Significance and Claims
The paper claims that the primary source of performance gain is the model's ability to attribute decisions to whole-fruit regions rather than local edges. By expanding the receptive field via dilated convolutions before dynamic routing, the network can resolve part-whole spatial agreements even in complex, occluded, and multi-object scenarios. The authors assert that this approach provides post-hoc evidence that the performance gains are not dataset artifacts but stem from a more robust representation of spatial structure.

The work highlights that while standard CNNs are limited by their pooling mechanisms in "in-the-wild" scenarios, capsule networks enhanced with dilated convolutions and rigorous hyper-parameter tuning can achieve superior accuracy with significantly fewer parameters. The authors acknowledge limitations, including unquantified label noise in the web-collected PD-19 dataset and the computational cost of routing, which currently constrains edge deployment. Future work is suggested to focus on quantitative localization metrics for explainability, adversarial pre-training, and few-shot learning for unseen cultivars.

More like this