💻 computer science

Finding NeMO: A Geometry-Aware Representation of Template Views for Few-Shot Perception

The paper introduces NeMO, a novel object-centric representation that enables few-shot detection, segmentation, and 6DoF pose estimation of unseen objects from RGB images by encoding sparse template views into a learned UDF, achieving state-of-the-art results on the BOP benchmark without requiring camera parameters or retraining.

Original authors: Sebastian Jung, Leonard Klüpfel, Rudolph Triebel, Maximilian Durner

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sebastian Jung, Leonard Klüpfel, Rudolph Triebel, Maximilian Durner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot to recognize a specific, strange-looking coffee mug that it has never seen before. Usually, to do this, you'd need to give the robot a perfect 3D blueprint (a CAD model) of that mug. But what if you don't have the blueprint? What if you only have a few photos of the mug taken from different angles on your phone?

This paper introduces a new method called NeMO (Neural Memory Object) that solves this problem. Think of NeMO as a "smart digital memory card" for an object.

Here is how it works, broken down into simple steps:

1. The "Memory Card" Creation (The Encoder)

Imagine you take a few photos of a new object (like that coffee mug) from different angles. You feed these photos into the NeMO system.

What it does: Instead of just storing the pictures, the system analyzes them to build a 3D "skeleton" of the object. It creates a cloud of points that represents the object's shape, texture, and features.
The Magic: This "skeleton" is stored in its own little coordinate system. It doesn't care about where the camera was or how the object was rotated; it just knows what the object is.
The Analogy: Think of this like taking a handful of photos of a friend and creating a 3D hologram of them in your mind. You don't need a blueprint of their face; you just need a few good pictures to build a mental model.

2. The "Search and Match" (The Decoder)

Now, imagine the robot is in a messy room (a "cluttered scene") and sees a new photo. It needs to find that specific coffee mug.

What it does: The robot takes the "memory card" (the NeMO) it created earlier and compares it to the new photo.
The Result: The system instantly tells the robot:
- Where the object is (Detection).
- What shape it is (Segmentation), even if part of it is hidden behind something else.
- How it is positioned in space (6DoF Pose), meaning exactly how it is tilted and rotated.
- What the surface looks like (Reconstruction), essentially guessing the full 3D shape even if the camera only sees a part of it.

3. Why This is Special

Most current AI systems are like students who memorize a specific textbook. If the test question changes slightly, they get confused. They usually need to be "retrained" (studied again) for every new object.

NeMO is different because it outsources the knowledge.

No Retraining: You don't need to teach the robot's brain (the neural network) anything new. You just give it a new "memory card" (NeMO) for the new object. The brain stays the same; only the memory changes.
Efficiency: Once the "memory card" is made, it's very fast to use. It doesn't matter if you used 5 photos or 50 photos to make the card; the robot uses the card at the same speed.
No Blueprints Needed: It works without a 3D CAD model. It learns the shape directly from real photos.

What the Paper Actually Proved

The authors tested this system on various datasets (collections of images used for testing AI). They showed that:

It can find and identify objects it has never seen before, even in messy, cluttered environments.
It can estimate the object's position and orientation very accurately.
It can even "guess" the full 3D surface of the object, allowing for reconstruction.
It performs as well as, or better than, other top methods that require 3D models or extensive retraining.

The Limitations (What it can't do yet)

The paper admits the system isn't perfect yet.

Symmetry: If an object looks the same from all sides (like a perfect sphere or a symmetrical cup), the system sometimes gets confused about which way is "up."
Textureless Objects: If an object is completely smooth and has no patterns (like a plain white ball), it's harder for the system to figure out the shape because it relies on visual features.
Multiple Objects: If there are two identical mugs in the picture, the system might merge them into one big blob instead of seeing them as two separate items.

In short, NeMO is a way to give a robot a quick, flexible "memory" of a new object using just a few photos, allowing it to recognize and interact with that object immediately without needing a 3D blueprint or a long training session.

Technical Summary: Finding NeMO

Problem Statement

The paper addresses the challenge of few-shot object perception for objects unseen during training. While model-based approaches (utilizing 3D CAD models) have achieved strong performance in detection, segmentation, and 6DoF pose estimation, they are impractical in many real-world scenarios where 3D models are unavailable. Conversely, existing model-free methods often suffer from poor scalability with the number of template views, rely on pairwise local comparisons without joint reasoning, or require extensive retraining and pre-processing to onboard new objects. The authors aim to develop a system that can rapidly adapt to novel objects using only a set of unordered RGB images, without requiring camera calibration parameters, 3D models, or retraining the core network weights.

Methodology: Neural Memory Object (NeMO)

The proposed solution is an encoder-decoder architecture centered around a novel object-centric representation called Neural Memory Object (NeMO).

1. NeMO Encoder

The encoder processes a set of $K$ unordered RGB template images ( $I = \{I_i\}$ ) depicting a target object.

Feature Extraction: A Vision Transformer (ViT) extracts patch-wise features from all template images.
Multi-View Encoding: A multi-view encoder, utilizing cross- and self-attention mechanisms, updates these features. One image is designated as an "anchor" ( $I_A$ ) to define the initial object orientation, while others interact with it to aggregate multi-view information.
Geometric Mapping: To incorporate 3D geometry, the system samples a set of 3D points ( $Q$ ) within a normalized volume. A Multi-Layer Perceptron (MLP) encodes these points into feature vectors. These point features are fused with the updated image features via a Geometric Mapping Block (using Transformer Decoders), allowing 3D points to attend to object-specific 2D visual cues.
Surface Reconstruction & Representation: The network learns an Unsigned Distance Field (UDF) to predict the distance from the initial 3D points to the estimated object surface. The final NeMO representation ( $\chi$ ) is a sparse, continuous point cloud where each point consists of an estimated 3D surface coordinate ( $s_i$ ) and its corresponding feature vector ( $\hat{f}^Q_i$ ).
Key Property: The object-specific information is disentangled from the decoder's parameters. The NeMO acts as a "memory" that can be transformed (rotated, translated, scaled) independently of the network weights.

2. NeMO Decoder

The decoder takes a query image ( $I_q$ ) and the pre-computed NeMO representation ( $\chi$ ) to generate dense predictions.

Feature Fusion: Query image features (extracted via ViT) are updated by attending to the NeMO features using cross- and self-attention.
Dense Prediction Heads: Using a DPT (Dense Prediction Transformer) backbone with multiple output heads, the decoder generates:
- Modal and Amodal Segmentation Masks: Predicting both visible and occluded parts of the object.
- 2D-3D Correspondence (Pointmap): A dense map linking 2D pixels in the query image to 3D surface points defined in the NeMO's coordinate system.
- Confidence Map: An associated confidence score for the 2D-3D mapping.
Pose Estimation: The 6DoF pose is estimated by filtering the pointmap based on confidence and applying RANSAC with Perspective-n-Point (PnP) algorithms.

3. Training Strategy

The encoder and decoder are trained jointly end-to-end on a large-scale synthetic dataset generated using BlenderProc.

Loss Functions: The training minimizes Euclidean distance between predicted and ground-truth surface points, segmentation losses (Dice and BCE), and a confidence-weighted L1 loss for 2D-3D correspondences.
Data Augmentation: During training, the NeMO point cloud is randomly rotated, translated, and scaled before being passed to the decoder. This forces the decoder to learn a coordinate system independent of the anchor image, ensuring robustness to the NeMO's orientation.
Offline/Online Separation: The computationally heavy encoding of template views is performed offline. The inference time depends only on the query image and the pre-computed NeMO, making it invariant to the number of input templates.

Key Contributions

NeMO Representation: A geometry-sensitive, compact, and continuous point-cloud representation of template views that encapsulates both semantic and geometric features without requiring camera parameters or 3D models.
Unified Architecture: A single encoder-decoder network capable of performing detection, segmentation, and 6DoF pose estimation for both model-free (RGB templates only) and model-based (CAD models available) settings.
Scalability and Efficiency: The method decouples object information from network weights, enabling quick object onboarding without retraining. The inference runtime and memory consumption remain constant regardless of the number of template images used to generate the NeMO.
New Dataset: The authors contribute a diverse, object-centric synthetic dataset mimicking realistic cluttered scenes to support research in this domain.

Experimental Results

The method was evaluated on the BOP benchmark across various datasets (HOPEv2, HANDAL, T-LESS, TUD-L, YCB-V) for model-free and model-based settings.

Model-Free Detection: The method achieved State-of-the-Art (SOTA) results on the HOPEv2 and HANDAL datasets, outperforming previous best methods by 2.7 and 1.9 percentage points in Average Precision (AP), respectively. Notably, it is the first to predict amodal segmentations in a model-free setting using a single network.
Model-Free Pose Estimation: The approach achieved SOTA results on the HANDAL dataset when using NeMO detections, performing on par with other methods when using alternative detections.
Model-Based Perception: In model-based settings (where CAD models are available at inference but not training), the method performed competitively, ranking 6th on YCB-V and achieving SOTA on TUD-L for detection. It demonstrated high recall on TUD-L and YCB-V for pose estimation without refinement, though it struggled with the symmetric, textureless objects in T-LESS.
Ablation Studies: Experiments showed that performance scales with the number of template images (saturating around 32 views) and the number of NeMO points (saturating around 500 points). The system demonstrated rotation-equivariance, where rotating the NeMO input resulted in corresponding rotations in the predicted pointmaps.

Significance and Claims

The paper claims that NeMO offers a versatile solution for model-free few-shot perception, effectively bridging the gap between the flexibility of model-free methods and the geometric precision of model-based approaches. By outsourcing object information into a NeMO, the system allows for:

Rapid Onboarding: New objects can be integrated without retraining the network.
Multi-Stage Pipelines: The ability to detect and segment objects in cluttered scenes allows for cropping regions of interest for subsequent processing.
Scalability: The separation of encoding and decoding ensures that adding more template views does not increase inference latency.

The authors acknowledge limitations, particularly regarding symmetric and highly textureless objects (as seen in the T-LESS dataset), attributing this to current training procedures and dataset biases rather than the representation itself. They conclude that their work stimulates discussion on decoupling object knowledge from model weights to advance few-shot perception.