💻 computer science

Fishbone: From One 3D Asset to a Million Controllable Edits

The paper introduces Fishbone, a unified rib-spine representation that transforms a single 3D mesh into a million controllable edits by enabling real-time parametric deformation, reduced-space dynamics, and animation through a geometrically aware structure inspired by natural creatures.

Original authors: Yumeng He, Xiaoying Wang, Peihao Li, Yanjia Huang, Joe Masterjohn, Jiajun Wu, Leonidas Guibas, Yin Yang, Ying Jiang, Chenfanfu Jiang

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yumeng He, Xiaoying Wang, Peihao Li, Yanjia Huang, Joe Masterjohn, Jiajun Wu, Leonidas Guibas, Yin Yang, Ying Jiang, Chenfanfu Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a 3D digital statue of a dragon, a chair, or a plant. Right now, if you want to change its shape—say, make the dragon's tail thicker or bend the chair's back—you usually have to do one of two difficult things:

The Sculptor's Way: Manually push and pull thousands of tiny dots (vertices) that make up the model. This is slow, tedious, and requires expert skill.
The Puppeteer's Way: Build a complex skeleton inside the model and manually attach strings (rigging) to every part. This takes hours of setup for every single new object.

Fishbone is a new tool that solves this by automatically finding the "skeleton" and "ribs" inside any 3D shape, just by looking at it. It turns a messy, complex 3D model into something as easy to control as a fish skeleton.

Here is how it works, using simple analogies:

1. The "Fishbone" Structure

The authors noticed that in nature, creatures like fish have a central spine (the backbone) that controls the overall curve and length, and ribs (the side bones) that control the width and thickness of the body at specific points.

Fishbone automatically builds this same structure inside any 3D object you give it:

The Spine: It finds a smooth line running through the very center of the object, like a backbone. This controls global movements like bending, twisting, or stretching the whole object.
The Ribs: It slices the object into cross-sections (like cutting a loaf of bread) and creates loops around the object at those slices. These ribs control local details, like making the waist thinner, the head wider, or changing the shape of a cross-section.
The Skin: It creates a "map" (called skinning weights) that connects the surface of the object to these internal bones. When you move a bone, the skin follows smoothly, just like real flesh moving over a skeleton.

The Magic: You don't have to build this skeleton yourself. The computer does it automatically in seconds, even for weird, free-form shapes like a crumpled piece of paper or a complex robot arm.

2. How You Control It

Once the Fishbone is built, you can edit the 3D model using simple sliders, similar to editing a photo but in 3D:

Rib Controls: Want to make a plant stem thicker? You grab a "rib" and scale it up. Want to twist a lamp shade? You rotate a rib.
Spine Controls: Want to bend a neck? You bend the spine. Want to stretch a tail? You pull the spine longer.

Because the system understands the geometry, it knows exactly how to stretch the skin without tearing it or making it look weird. It's like having a magic wand that knows exactly how to reshape the clay without you needing to be a master sculptor.

3. Making It Move (Animation and Physics)

The paper also shows that this Fishbone structure is perfect for making things move realistically without heavy computing power.

Reduced Simulation: Instead of calculating the physics for millions of tiny dots on the surface (which is slow), the computer only calculates the physics for the few "spine points." It then projects those movements back onto the surface.
The Analogy: Imagine a wind blowing through a forest. Instead of calculating how every single leaf moves, you just calculate how the tree branches sway. The leaves follow naturally because they are attached to the branches. Fishbone does this for 3D models, allowing for real-time effects like a bird's wings drooping under gravity or a plant swaying in the wind.

4. What They Did With It

The researchers didn't just build the tool; they used it to create a massive dataset of 136,000 3D objects (Fishbone-136K) that already have these skeletons built-in. They demonstrated three main uses:

Generating New Shapes: They trained an AI to look at a picture and instantly generate a 3D model plus its Fishbone skeleton, so it's ready to be edited immediately.
Robot Learning: They used the tool to create thousands of slightly different versions of objects (e.g., a cup that is slightly taller, wider, or twisted). They fed these to a robot learning to grab things. The robot learned to grab any cup, not just the specific one it was trained on, because it saw so many variations.
AI Agents: They built a system where you can type a command like "Make the four legs thicker," and an AI agent automatically finds the right ribs and thickens them, checking its own work to make sure it looks right.

Summary

Fishbone is like giving every 3D object an automatic, invisible skeleton and ribcage. This turns complex, hard-to-edit 3D models into easy-to-control objects that can be bent, twisted, and animated instantly. It removes the need for manual rigging, works on any shape, and opens the door for robots to learn better and for AI to generate controllable 3D worlds.

Technical Summary: Fishbone: From One 3D Asset to a Million Controllable Edits

1. Problem Statement

The availability of large-scale, diverse, and controllable 3D assets is a critical bottleneck for computer graphics, embodied AI, robotics, and interactive content creation. While 3D scanning and artist-authored assets exist, creating diverse variations of these assets remains challenging due to the high cost of manual modeling and rigging.

Existing solutions for shape deformation face significant limitations:

Manual Rigging: Traditional methods (cage-based coordinates, skeleton-driven animation, free-form deformation lattices) require substantial manual setup (e.g., designing cages, placing skeletons) that does not scale to heterogeneous collections of assets.
Sparse Control Limitations: Data-driven methods relying on sparse user inputs (2D sketches, projected points) often suffer from depth uncertainty and view-dependent artifacts, limiting precision on fine-grained local geometry.
Category Specificity: Recent 3D control methods (neural cages, learned flow fields) often require category-specific training data and degrade on out-of-distribution shapes.
Parametric Rigidity: Parametric modeling offers fine-grained control but relies on well-defined structures and clean topology, which breaks down on free-form geometry.

The core challenge is to construct a geometry-derived, low-dimensional control representation for general shapes that enables predictable parametric deformation without manual rigging, category-specific training, or predefined shape structures.

2. Methodology: The Fishbone Framework

The authors propose Fishbone, a unified framework that constructs a coupled rib-spine representation directly from an input mesh. Inspired by natural creatures where a central spine governs global shape and cross-sectional ribs control local variation, Fishbone decouples local from global shape variation through three coupled components: Ribs ( $R$ ), a Spine ( $S$ ), and Skinning Weights ( $W$ ).

2.1 The Fishbone Representation ( $F = (R, S, W)$ )

Ribs ( $R$ ): A set of ordered polylines representing cross-sectional iso-contours. They encode local shape profiles (thickness, orientation, cross-sectional variation).
Spine ( $S$ ): A smooth centerline threading through the volumetric centers of the ribs. It governs the global trajectory, bending, twisting, and stretching of the shape. The spine is stored as a shared-node tree to handle branching structures (e.g., limbs separating from a torso) without duplicating junction points.
Skinning Weights ( $W$ ): Sparse matrices ( $W_r$ for ribs, $W_s$ for spine) that bind mesh vertices to the control structures using Gaussian radial basis functions, enabling smooth propagation of deformations.

2.2 Mesh-to-Fishbone Pipeline

The framework automatically extracts this structure from general triangle meshes without manual intervention:

Automatic Root Selection: For each mesh part, a dominant geometric direction is estimated from the axis-aligned bounding box to initialize the root for geodesic computation.
Geodesic Field Computation: An adaptive heat method solves a short-time heat diffusion process followed by a Poisson equation to generate a smooth, monotonically increasing scalar field ( $\phi$ ) from the root.
Rib Extraction: Iso-contours of $\phi$ at multiple geodesic levels are extracted as ribs. The number of ribs ( $K$ ) is adaptively determined based on the part's relative extent to ensure consistent spacing. Topology-aware propagation (via BFS on face-adjacency graphs) handles branching and disconnected components.
Spine Construction: A geometry-aware center-finding algorithm extracts a representative spine point for each rib by maximizing a score combining local flatness, geometric centering, and parent-child continuity. These points form a branching spine tree.
Skinning Weight Computation: Adaptive Gaussian weights are computed to bind vertices to the nearest rib edges and spine points.

2.3 Deformation Primitives

The representation supports nine parametric deformation primitives:

Rib-Driven (Local): Uniform/anisotropic scaling, translation, rotation, local dragging, and cross-section reshaping. These manipulate local profiles while preserving global structure.
Spine-Driven (Global): Stretching, bending, and twisting. These manipulate the global trajectory and orientation.
Bidirectional Coupling: Deformations can be chained. A rib edit updates the mesh and subsequently the spine; a spine edit updates the mesh and the ribs, maintaining the coupled structure without re-extraction.

2.4 Reduced-Space Dynamics and Animation

Reduced Simulation: The spine serves as low-dimensional simulation coordinates. The full mesh state is lifted from the spine using the skinning matrix. The system simulates physics (elasticity, damping, external forces like wind/gravity) on the sparse spine points, enabling real-time, physically plausible secondary motion (e.g., plant sway, jiggle) without simulating every vertex.
Keyframe Animation: Users can author sparse target poses via deformation primitives. Intermediate frames are generated by interpolating between keyframes in the low-dimensional parameter space (mesh vertices, ribs, and spine points).

3. Key Contributions

Unified Fishbone Representation: A novel coupling of geometry-derived ribs, a smooth spine, and Gaussian-weighted skinning. This enables intuitive, efficient local and global parametric deformations and supports reduced-space dynamics and animation within a single structure.
Automatic Mesh-to-Fishbone Framework: A pipeline that extracts rib-spine structures from general meshes without manual rigging, category-specific training, or predefined templates. This pipeline is used to construct the Fishbone-136K dataset, containing part-level meshes with corresponding ribs, spines, and skinning structures.
Versatile Applications and Evaluation: The framework is demonstrated across 3D generation (jointly generating meshes and control structures), interactive editing, robot learning (data augmentation for grasping), and agentic generation. Extensive experiments validate its effectiveness, efficiency, and versatility.

4. Results and Performance

Runtime Efficiency:
- Preprocessing: The extraction pipeline runs in a median of 58.0 seconds per mesh (mean 84.5s) on a 2,565-mesh dataset. Runtime scales roughly linearly with vertex count.
- Deformation: Interactive edits are performed in milliseconds (median ~1ms for most primitives, ~12.7ms for twisting) on meshes with up to 1.6M vertices.
- Dynamics: Reduced-space simulation achieves real-time performance (median 1.1ms per substep for physics, ~3ms with displacement lift). The reduction ratio ( $K/N$ ) is typically around 0.007%, simulating only a tiny fraction of the original vertices.
Quality and Accuracy:
- Deformation: Compared to ARAP and WIR3D, Fishbone achieves lower Laplacian smoothness deviation (fewer artifacts) and lower ARAP energy (better rigidity preservation) with significantly lower preprocessing time (seconds vs. hours).
- 3D Generation: In image-conditioned generation, the joint prediction of meshes and Fishbone structures outperforms baselines (PartCrafter, OmniPart, PartPacker) in Chamfer Distance and F-Score, producing cleaner geometries with fewer floating artifacts.
- Robot Learning: Fine-tuning a grasping policy (DexGraspNet 2.0) on Fishbone-augmented data significantly improves grasp success rates on unseen, in-the-wild objects in both single and multi-object scenes.
- Ablation Studies: Removing components (heat method, automatic root selection, adaptive level count, branching tree handling) leads to fragmented ribs, misaligned spines, or unstable deformations, confirming the necessity of each design choice.

5. Significance and Claims

The paper claims that Fishbone provides a scalable, automatic, and geometry-aware solution for 3D asset manipulation. By deriving a low-dimensional control structure directly from the mesh geometry, it bridges the gap between the precision of parametric modeling and the flexibility of free-form shapes.

The significance lies in its ability to:

Eliminate the need for manual rigging and category-specific training for diverse assets.
Enable predictable and controllable edits (local and global) that are difficult to achieve with sparse 2D controls or non-parametric neural fields.
Support real-time reduced-space dynamics and animation using the same control structure used for editing, facilitating applications in robotics (data augmentation) and interactive content.
Serve as a foundational representation for agentic systems, allowing LLMs/VLMs to reason about and execute complex mesh edits (e.g., "thicken the legs") through stable, semantic handles.

The authors acknowledge limitations, noting the current reliance on triangle meshes and the manual effort required for authoring long keyframe sequences, suggesting future work in integrating with other 3D representations (point clouds, Gaussian Splatting) and learning-based keyframe generation.