💻 computer science

MeshFM: 2D Features Are All You Need for 3D Shape Understanding

MeshFM is an efficient, feedforward framework that distills rich 2D features from visual foundation models into 3D representations without requiring 3D annotations or inference-time optimization, achieving performance on par with 3D-supervised methods across various downstream tasks.

Original authors: Jinfan Zhou, Richard Liu, Itai Lang, Rana Hanocka

Published 2026-07-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jinfan Zhou, Richard Liu, Itai Lang, Rana Hanocka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand the world. For a long time, scientists had to teach robots about 3D objects—like chairs, cats, or cars—by showing them millions of 3D models and manually labeling every single part. It was like teaching a child to recognize a dog by showing them a thousand plastic dog statues and pointing out the ears, tail, and paws on each one. It was slow, expensive, and required a massive amount of human effort.

Then, a revolution happened in the world of 2D images. Huge "foundation models" were trained on the entire internet, learning to recognize objects, textures, and even specific parts of things just by looking at flat pictures. These models became so smart that they could understand a photo of a dog without ever seeing a 3D model of one. The big question for scientists became: Can we take this incredible 2D knowledge and "lift" it into the 3D world? Can we teach a robot to understand a 3D chair just by showing it 2D photos, without needing to manually label the 3D shape? This is the challenge that the paper "MeshFM" tackles.

The researchers behind MeshFM, a team from the University of Chicago, propose a clever new way to do this. They argue that we don't need to build special, complicated 3D brains for every single task. Instead, they suggest that if we can clean up the 2D knowledge properly, it is enough to understand 3D shapes perfectly.

Here is how they did it, using a story about a master painter and a student:

The Problem: The Blurry Copy
Imagine a master painter (the 2D foundation model) who can see a 3D object from every angle and describe its parts perfectly. But when they try to paint a 3D sculpture based on their 2D sketches, they use a technique that sometimes smears the paint. If the painter is looking at a chair leg next to the seat, the colors might bleed into each other, making it hard to tell where the leg ends and the seat begins. This is called "feature bleeding" or "aliasing." Previous methods tried to fix this by having a student (a 3D network) slowly learn from the master, but the student kept getting confused by the smudges, or the process was so slow it took hours to learn just one object.

The Solution: The Two-Stage Training
The MeshFM team came up with a two-step plan to fix this.

Stage 1: The Master's Clean Sketch. First, they took the 2D master painter and asked it to look at a 3D object from many different angles. But here's the trick: they didn't just let the paint smear. They used a super-precise tool (called SAM, a segmentation model) to act like a sharp knife, cutting away the blurry edges where the paint bled. They cleaned up the "outlier" pixels that didn't belong, ensuring that the color of the chair leg stayed distinct from the seat. They then used this clean, sharp information to create a perfect "teacher" map of the 3D object. This map was so good it knew exactly where every part was, even if the object was rotated.
Stage 2: The Fast Student. Next, they trained a fast, feedforward student network. Instead of slowly learning from scratch, this student looked at the clean "teacher" maps created in Stage 1. The student learned to look at a 3D shape (represented as a cloud of points) and instantly predict the same clean, sharp features. To make sure the student didn't get confused if the object was turned upside down or sideways, they spun the objects around in every possible direction during training. This made the student "rotation-robust," meaning it could recognize a chair whether it was standing up, lying down, or spinning in the air.

The Big Discovery
The most exciting part of their finding is that this student, trained only on 2D photos and without any human labels for 3D shapes, turned out to be incredibly smart. They tested it on three different jobs:

Segmentation: Breaking a shape into parts (like separating a chair's legs from its back).
Correspondence: Finding the matching point on two different shapes (like finding the "nose" on a cat and the "nose" on a dog, even if they look different).
Deformation: Bending and twisting a shape (like squishing a 3D model of a character).

In every test, MeshFM performed just as well as, or even better than, the best existing methods that were trained specifically for those tasks using expensive 3D data. Even when they rotated the objects randomly, MeshFM kept working perfectly, while the other methods got confused and failed.

What They Ruled Out
The authors explicitly argue against the idea that we need specialized, task-specific 3D networks for every job. They suggest that the reason previous methods struggled wasn't because 2D features were weak, but because the way they were transferred to 3D was messy. By cleaning up the transfer process, they showed that 2D features are actually "all you need" for 3D understanding. They also ruled out the idea that you need to optimize (slowly tweak) the model for every single new object; MeshFM works instantly in a single forward pass.

How Sure Are They?
The team is quite confident in their results, backed by extensive experiments on standard datasets. They showed that their method works on a wide variety of objects, from animals to electronics. However, they do admit a small limitation: because their "master painter" (the 2D model they used) sometimes struggles to tell left from right, their 3D model occasionally swaps left and right parts (like confusing a left shoe for a right one). But they note that as 2D models get better, their 3D model will get better too.

In short, MeshFM suggests that we don't need to reinvent the wheel for 3D. By taking the amazing knowledge we already have from 2D internet photos, cleaning it up, and teaching a fast network to use it, we can give computers a deep, flexible understanding of the 3D world without needing millions of human labels.

Technical Summary: MeshFM

Problem Statement

The landscape of 3D shape understanding has been hindered by the reliance on specialized 3D networks and the scarcity of 3D annotated data. While large-scale 2D foundation models (e.g., DINO, CLIP, SAM) have demonstrated rich, general-purpose visual understanding, transferring these capabilities to 3D remains challenging. Existing approaches generally fall into two categories, both with significant limitations:

Distillation via Optimization: Methods that render 2D views, extract features, and back-project them onto 3D shapes via per-shape optimization (e.g., Diff3F, DFD). While effective, these are computationally expensive and impractical for real-time inference.
Indirect 2D Priors: Methods that train task-specific 3D networks using 2D signals (e.g., PartField, DenseMatcher). These often require 3D supervision for fine-tuning, suffer from data bottlenecks, and produce features that are not generalizable beyond their specific target tasks.

Furthermore, standard distillation techniques suffer from feature aliasing and "feature bleeding." Due to the patch-based nature of Vision Transformer (ViT) encoders, features at object boundaries often blur, causing background semantics to bleed into the object or distinct parts to lose sharp boundaries. This results in noisy 3D feature fields that degrade downstream performance.

Methodology

The authors propose MeshFM, an efficient feedforward framework that extracts rich, general-purpose features from 3D inputs by distilling 2D foundation model features into 3D. The method operates in a two-stage training pipeline that requires no 3D annotations and no task-specific fine-tuning.

Stage 1: Refined Teacher Distillation

The first stage focuses on creating high-quality "teacher" 3D feature fields ( $\Phi$ ) for each training shape by distilling 2D signals.

Barycentric Feature Distillation: Following DFD, the method renders multi-view images of a mesh, extracts 2D feature maps from a foundation model (e.g., DINOv2), and back-projects them to the 3D surface using barycentric coordinates. This decouples the supervision signal from the mesh triangulation.
Feature Aliasing Correction (SAM-based): To address feature bleeding, the authors introduce a refinement strategy leveraging the Segment Anything Model (SAM).
- For each rendered view, SAM generates disjoint segmentation masks.
- Within each mask, the method assumes pixels belong to a semantically consistent part. It computes a representative feature vector (channel-wise median) for the segment.
- Pixels identified as "outliers" (where the raw feature deviates significantly from the segment median) are replaced with the representative median feature.
- This process effectively "wipes" bleeding features, producing a refined, boundary-aware ground truth feature field.

Stage 2: Rotation-Robust Feedforward Prediction

The second stage trains a feedforward network ( $\Psi$ ) to regress the optimized teacher fields directly from a point cloud input, eliminating the need for test-time optimization.

Architecture: The network uses a PVCNN encoder to process the input point cloud, projecting features onto three axis-aligned planes (XY, XZ, YZ) to form a triplane representation. These triplanes are processed by a 2D CNN and a Transformer to capture global context, then upsampled to yield the final feature tensor.
Training Objective: The network minimizes the L2 loss between its predicted features and the target features sampled from the frozen Stage 1 teacher fields.
SO(3) Rotation Augmentation: To ensure the model handles objects in arbitrary orientations (crucial for "in-the-wild" applications), the authors apply aggressive SO(3) rotation augmentation during training. The maximum rotation angle is linearly increased to $2\pi$ over the training iterations.

Key Contributions

Feedforward 3D Feature Extraction: MeshFM is the first method to predict rich, general-purpose 3D features in a single forward pass without per-shape optimization or 3D supervision.
Refined Distillation via SAM: The introduction of a SAM-based feature refinement strategy successfully mitigates feature aliasing and bleeding inherent in ViT-based encoders, producing sharper, more semantically consistent 3D feature fields.
Rotation Robustness: By training with full-range SO(3) augmentation, the model achieves robustness to extreme rotations, a common failure point for prior task-specific 3D models.
Zero-Shot Generalization: The learned features are demonstrated to be applicable to diverse downstream tasks (segmentation, correspondence, deformation) without any task-specific fine-tuning.

Experimental Results

The authors evaluate MeshFM on three primary downstream tasks, comparing it against state-of-the-art (SOTA) baselines (PartField, DenseMatcher, Diff3F) using zero-shot inference.

Part Segmentation: On the PartObjaverse-Tiny and PartNetE datasets, MeshFM achieves mean Intersection-over-Union (mIoU) scores on par with or superior to SOTA baselines. Notably, while baselines like PartField degrade significantly under rotation, MeshFM maintains high performance, demonstrating superior rotation robustness.
Dense Correspondence: Evaluated on TOSCA and DenseCorr3D, MeshFM performs competitively with SOTA methods on canonical shapes and significantly outperforms them on rotation-augmented shapes. Baselines exhibit strong canonical coordinate bias, failing to align shapes when rotated, whereas MeshFM maintains consistent correspondence.
Classification & Deformation:
- Classification: A lightweight MLP trained on MeshFM features achieves higher accuracy on the Manifold40 dataset compared to features from other methods, particularly under rotation.
- Deformation: The features enable smooth, semantic-aware mesh deformations via affine transformations, successfully isolating semantic parts regardless of initial orientation.

Ablation Studies:

Removing the SAM-based aliasing correction results in a measurable drop in segmentation performance, confirming the necessity of the refinement step.
The method is robust to the choice of the underlying 2D foundation model (tested with DINOv2, DINOv3, RADIO, and SAM2), though SAM features showed slightly lower performance due to view inconsistency.

Significance and Claims

The paper posits that the reliance on task-specific 3D heads is largely a symptom of imperfect feature distillation. The authors claim that 2D features, when properly distilled and refined, are sufficient for comprehensive 3D shape understanding.

The significance of MeshFM lies in its ability to:

Democratize 3D Understanding: Eliminate the need for expensive 3D annotations and per-shape optimization.
Bridge the 2D-3D Gap: Successfully transfer the rich semantic priors of 2D foundation models to 3D geometry in a scalable, feedforward manner.
Enable Robust Zero-Shot Applications: Provide a single, general-purpose representation that works effectively across segmentation, correspondence, and deformation tasks, even for arbitrarily oriented objects.

The authors acknowledge that the method inherits limitations from the supervising 2D models, such as difficulty distinguishing left-right symmetric parts (e.g., chair legs) due to the semantic reasoning nature of models like DINOv2. However, they argue that as 2D foundation models improve, the capabilities of MeshFM will naturally improve as well.