💻 computer science

Instruct-Particulate: Scaling Feed-Forward 3D Object Articulation with Kinematic Control

The paper introduces Instruct-Particulate, a model that leverages kinematic specifications and a large-scale heterogeneous dataset of over 150,000 objects to significantly improve the generalization and scalability of reconstructing articulated 3D objects from meshes or real-world images.

Original authors: Ruining Li, Yuxin Yao, Matt Zhou, Chuanxia Zheng, Christian Rupprecht, Joan Lasenby, Shangzhe Wu, Andrea Vedaldi

Published 2026-06-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruining Li, Yuxin Yao, Matt Zhou, Chuanxia Zheng, Christian Rupprecht, Joan Lasenby, Shangzhe Wu, Andrea Vedaldi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a digital 3D model of a toy robot or a kitchen appliance. Right now, this model is just a solid, frozen lump of plastic—it can't move its arms, open its doors, or press its buttons. It's like a statue.

The paper introduces a new AI tool called INSTRUCT-PARTICULATE that acts like a "digital mechanic." Its job is to take that frozen 3D statue and figure out exactly how to cut it into moving pieces, then tell the computer how those pieces should swing, slide, or rotate.

Here is how it works, broken down into simple concepts:

1. The Problem: Too Many Puzzles, Not Enough Answers

Previously, teaching computers to understand moving parts was like trying to solve a jigsaw puzzle with missing pieces. There weren't enough examples (data) of 3D objects with their moving parts labeled. Because the AI didn't see enough examples, it struggled to guess how a new, strange object (like a weird coffee machine) should move.

2. The Solution: A "Teacher" with a Cheat Sheet

The researchers realized that while they didn't have enough labeled 3D puzzles, they had millions of unlabeled 3D models and a very smart "teacher" AI (a Vision-Language Model) that could look at a picture and say, "That's a lid, that's a hinge, and that's a button."

They built a system to use this "teacher" to label thousands of new 3D objects automatically. It's like hiring a robot assistant to go through a warehouse of toys, point out every moving part, and write down the rules for how they move. This gave them a massive library of over 150,000 examples to train their model.

3. The Magic Trick: "Instruction-Based" Learning

Here is the clever part. Sometimes, a single object can be broken down in different ways. For example, a drawer could be one big piece, or it could be split into a handle and a box. If you just ask the AI "How does this move?", it might get confused and give a messy, average answer.

INSTRUCT-PARTICULATE solves this by asking for instructions.

The Prompt: You can tell the AI exactly what you want. You can say, "Treat this as a box with a rotating lid," or "Treat this as a chair with a swivel base."
The Pointers: You can even point to a specific spot on the 3D model and say, "This part is the handle."

Think of it like giving a chef a recipe. Instead of guessing what to cook, you give them the specific ingredients and the steps. This stops the AI from getting confused and helps it produce a clean, precise result.

4. How It Works in Practice

The system takes a static 3D shape (like a mesh of a toaster) and a set of instructions.

It looks at the shape: It scans the surface of the object.
It listens to the instructions: It reads your text description or looks at the points you clicked.
It predicts the anatomy: It instantly figures out which pixels belong to the "lid," which belong to the "base," and where the "hinge" is.
It calculates the movement: It determines the exact axis (the invisible rod) the part rotates around and how far it can turn.

5. The Result: From Photo to Moving Toy

The paper shows that you can take a simple photo of a real-world object (like a microwave), turn it into a 3D model, and then use this tool to make it move.

Before: The microwave was a solid block.
After: The AI knows exactly where the door hinge is, how the door swings open, and where the buttons are. It can even handle complex objects like stand mixers or coffee machines that previous AI models failed to understand.

Summary

INSTRUCT-PARTICULATE is a tool that teaches computers to understand the "skeleton" of moving 3D objects. By using a massive amount of automatically labeled data and letting users give specific instructions (like "this is the handle"), it can turn static 3D models into realistic, moving assets that can be used for animation, gaming, or robot simulations. It's essentially a way to give "muscles and joints" to digital statues on command.

Technical Summary: INSTRUCT-PARTICULATE

Problem Statement

Reconstructing the articulated structure of 3D objects—decomposing them into kinematic parts with associated joint parameters—is critical for animation, gaming, and robotic simulation. However, progress in this domain has been bottlenecked by the scarcity of high-quality, annotated articulated 3D data. Existing methods rely on small, manually curated datasets (e.g., PartNet-Mobility), which limits their ability to generalize to novel object categories or diverse geometries. Furthermore, the inherent ambiguity in defining part granularity and semantics for a given 3D object often leads models trained on heterogeneous data to produce suboptimal, "averaged" results.

Methodology

The authors propose INSTRUCT-PARTICULATE, a feed-forward model designed to infer articulated structures from static 3D assets (meshes) guided by explicit kinematic instructions.

1. Data Construction and Scaling

To overcome data scarcity, the authors constructed a heterogeneous dataset of over 150,000 articulated 3D objects using three complementary strategies:

VLM-based Pseudo-Labeling: A pipeline utilizing Vision-Language Models (VLMs) to extract kinematic structures (part names, connectivity, joint types) from multi-view renderings of 27,000 synthetic 3D assets. The VLM generates segmentation masks which are unprojected to 3D.
Augmentation of Part-Segmented Models: For 120,000 assets with existing part decompositions (but lacking kinematic labels), the pipeline generates textual captions for parts. Crucially, to handle ambiguity (e.g., identical table legs), the system uses spatial cues only when a canonical orientation exists; otherwise, it assigns identical captions to semantically identical parts, relying on geometric point prompts for distinction.
Coding Agent Generation: 10,000 fully supervised articulated objects were generated using Articraft, a specialized 3D coding agent, providing precise joint parameters.

2. Model Architecture

The model employs a scalable encoder-decoder architecture that processes a 3D mesh (represented as a point cloud) and a kinematic condition $C$ . The condition $C$ includes:

A kinematic tree structure $K$ (parts and connectivity).
Text prompts ( $t_p$ ) describing each part.
Optional 3D point prompts ( $x_p$ ) indicating specific locations on parts.

Encoding:

Shape Tokens: Points sampled from the mesh are encoded with surface normals and features (via PartField) and compressed into latent shape tokens via cross-attention.
Query Tokens: Additional points sampled on the surface are encoded to predict part associations.
Part Tokens: Text prompts are encoded via CLIP and combined with point prompt embeddings.

Processing:
The tokens are processed through $B$ transformer blocks. Part tokens update via self-attention and cross-attention to shape tokens to infer articulation parameters. Query tokens cross-attend to both shape and updated part tokens to determine part membership. Notably, self-attention is omitted between query tokens to ensure speed and independence from the number of query points.

Decoding:

Part Segmentation: An MLP scores query-part pairs to assign surface points to specific parts.
Joint Motion Parameters: For each movable joint, the model predicts motion parameters (axis direction, pivot point, and motion bounds) in an over-parameterized manner. It predicts, for each query point, the closest point on the joint axis and the point's location at the joint's lower and upper limits. These per-query predictions are aggregated geometrically to recover a single shared motion axis and range for each joint.

Key Contributions

INSTRUCT-PARTICULATE Model: A state-of-the-art feed-forward model that accepts a static 3D object and kinematic instructions to output a fully articulated 3D object, including part segmentation and joint motion parameters.
Scalable Data Engine: A pipeline that leverages VLMs and coding agents to pseudo-label and generate a large-scale, diverse dataset of over 150k articulated 3D objects, significantly expanding beyond existing manually annotated collections.
Kinematic Conditioning: A novel formulation where explicit kinematic instructions (structure, text, and points) disambiguate the target articulated structure, allowing the model to learn from diverse annotations without averaging them into suboptimal outputs.
Image-to-3D Articulation: Demonstration of a practical pipeline where the model, combined with off-the-shelf 3D generators, can generate diverse, realistic articulated 3D assets directly from real-world images.

Experimental Results

The model was evaluated on the Lightwheel dataset, which contains 243 high-quality articulated objects across 14 categories, including those absent from prior datasets.

Quantitative Performance: INSTRUCT-PARTICULATE consistently outperformed eight existing baselines (including SINGAPO, PAct, PhysX-Anything, and URDF-Anything+) across all input modes (Image Only, Mesh, and Mesh + Kinematic).
- In the "Mesh + Kinematic" setting, it achieved a 97.3% part match precision and 95.9% recall, with a 0.821 gIoU for rest-pose segmentation and 0.757 gIoU for articulated geometry.
- It significantly reduced joint axis errors (Angle Error: 9.5°, Location Error: 0.015) compared to baselines.
Ablation Studies:
- Data: Models trained only on existing datasets (PartNet-Mobility, GRScenes) severely overfit and failed to generalize. The inclusion of the pseudo-labeled and coding-agent-generated data provided complementary gains in both segmentation and joint estimation.
- Conditioning: Removing kinematic conditioning led to a drastic performance drop, confirming that explicit instructions are necessary to resolve ambiguity in part granularity and semantics. Point prompts provided the largest performance gain among prompt types.
Qualitative Results: The model successfully recovered small articulated parts (buttons, knobs) and complex structures (stand mixers, coffee machines) where baselines failed. It also demonstrated the ability to follow specific kinematic prompts, such as distinguishing left vs. right drawers via text or handling 24 distinct button joints.

Significance and Claims

The paper claims that INSTRUCT-PARTICULATE represents a significant step forward in articulated object understanding by addressing the data scarcity bottleneck through scalable, VLM-driven data engineering. By introducing kinematic control, the model can target specific annotation granularities, enabling the use of heterogeneous training data that was previously unusable.

The authors assert that this approach enables the generation of diverse, physics-ready articulated assets directly from real-world images, bridging the gap between static 3D generation and dynamic simulation. The work is presented as a practical solution for creating simulation assets for embodied AI, moving beyond the limitations of small, manually annotated datasets to a regime of large-scale, category-agnostic learning.