💻 computer science

SARe: Structure-Aware Large-Scale 3D Fragment Reassembly

The paper proposes SARe, a structure-aware generative framework that combines a contact-graph-based assembly generator with an inference-time refinement stage to achieve robust, state-of-the-art 3D fragment reassembly, particularly in challenging large-scale scenarios with many fragments.

Original authors: Hanze Jia, Chunshi Wang, Yuxiao Yang, Zhonghua Jiang, Yawei Luo, Shuainan Ye, Tan Tang

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hanze Jia, Chunshi Wang, Yuxiao Yang, Zhonghua Jiang, Yawei Luo, Shuainan Ye, Tan Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are handed a shattered vase, but instead of a few big pieces, you have 50 tiny, irregular shards. Your goal is to glue them back together perfectly without knowing what the original vase looked like. This is the challenge of 3D Fragment Reassembly.

For a long time, computers have been terrible at this when the number of pieces gets high. They might get the first few pieces right, but as they try to add more, they start guessing wrong, and the whole structure collapses into a messy blob.

The paper introduces a new AI system called SARe (Structure-Aware Reassembly) that solves this by changing how the computer thinks about the problem. Here is the breakdown using simple analogies:

1. The Problem: The "Blind Puzzle"

Most previous AI methods tried to solve this like a blindfolded puzzle solver. They would guess where every piece goes based on how the edges might fit.

The Flaw: If the AI guesses that Piece A connects to Piece B, but it's actually wrong, that mistake messes up the placement of Piece C, which messes up Piece D, and so on. This is called a "cascading failure."
The Result: As the number of pieces grows (from 10 to 50), the success rate of old methods crashes because they are too afraid to make a structural guess.

2. The Solution: SARe (The "Architect + Inspector")

SARe doesn't just guess where pieces go; it acts like a smart architect who draws a blueprint while building, and then a rigorous inspector who checks the work before moving on.

It has two main stages:

Stage A: SARe-Gen (The Architect)

Instead of just guessing the final position of every piece, this part does two things simultaneously:

Builds the Shape: It predicts where every piece should go in 3D space.
Draws the Map: It explicitly predicts a "Contact Map" (a graph showing which pieces touch which).

The Analogy: Imagine you are assembling a Lego castle.

Old AI: Just tries to snap pieces together blindly until it looks like a castle.
SARe-Gen: While snapping pieces, it also draws a diagram saying, "The red tower must touch the blue wall here." It treats the connection points as a primary clue, not just a side effect. This prevents the "cascading failure" because the AI knows the rules of the structure before it fully builds it.

Stage B: SARe-Refine (The Inspector)

Sometimes, even the best architect makes a mistake on a tricky corner. SARe-Refine is a second pass that fixes these errors without starting over.

The Analogy: Imagine you are painting a mural.

You finish the first draft, but some parts look wobbly.
Instead of wiping the whole canvas clean, the Inspector looks at the parts that are already perfect (the "reliable substructures") and says, "Okay, these are solid. Lock them in place."
Then, it only resamples (re-paints) the uncertain, wobbly areas, using the locked-in parts as a guide.
Why it works: It stops the AI from "un-learning" the good parts while trying to fix the bad ones.

3. How It Learns (The "Frozen Brain")

To understand the shape of the shards, SARe uses a pre-trained "brain" (a frozen geometry encoder) that has already seen millions of 3D shapes.

The Trick: It doesn't try to learn how to recognize shapes from scratch. Instead, it asks this pre-trained brain: "Here is a specific spot on a shard; what does the local geometry look like?"
This allows the system to focus its energy on solving the puzzle rather than learning what a piece of pottery looks like.

4. The Results: Why It Matters

The authors tested SARe on three types of challenges:

Synthetic: Computer-generated breaks (easy).
Simulated Real: Scanned real objects that were digitally broken (medium).
Real World: Objects that were physically broken and scanned (hard).

The Outcome:

When the number of pieces was small, SARe was good.
When the number of pieces grew to 50 (a very hard scenario), SARe kept getting better, while other methods failed completely.
It achieved State-of-the-Art results, meaning it is currently the best in the world at this task.

Summary

Think of SARe as a master puzzle solver who doesn't just force pieces together.

It predicts the connections (the "who touches whom" map) before finalizing the shape.
It locks in the correct parts and only fixes the messy parts in a second round.
This allows it to solve massive, complex puzzles (like restoring ancient artifacts or helping robots assemble debris) that used to be impossible for computers.

1. Problem Definition

3D Fragment Reassembly involves recovering the rigid-body poses ($SE(3)$) of unordered fragment point clouds or meshes to reconstruct a complete 3D object.

Challenges: The problem becomes significantly harder as the number of fragments ( $K$ $K$ ) increases.
- Unknown Target: The global shape is unknown during inference.
- Weak Semantics: Fracture fragments often lack semantic cues and have irregular shapes.
- Cascading Failures: Existing end-to-end methods often fail at scale because they treat inter-fragment adjacency as an implicit byproduct. Inaccurate contact reasoning leads to structural errors that propagate and amplify through the global assembly.
Goal: To achieve robust, scalable reassembly that maintains global consistency even with high fragment counts (up to $K=50$ ) by explicitly modeling contact structures.

2. Methodology

The authors propose SARe (Structure-Aware Reassembly), a generative framework consisting of two main stages: SARe-Gen (Generation) and SARe-Refine (Refinement).

A. SARe-Gen: Structure-Aware Generation

Instead of directly regressing poses in $SE(3)$ space, SARe-Gen formulates the problem as conditional modeling in Euclidean space using Rectified Flow.

Query-Point Conditioning:
- Fragments are represented by a fixed budget of sparse query points sampled via Farthest Point Sampling.
- A frozen geometry encoder (ShapeVAE) extracts query-aligned local geometric tokens from the fragment surfaces.
- These tokens are conditioned on fragment identity embeddings and positional encodings, creating a point-aligned conditioning set $C$ .
Joint Prediction (The Core Innovation):
- The model uses a DiT-style transformer to denoise the assembled query points ( $x_0$ ) from a noise endpoint ( $x_1$ ).
- Explicit Structural Heads: Unlike prior work, SARe-Gen attaches lightweight heads to the transformer's intermediate layers to jointly predict:
  - Fracture-Surface Probabilities: Token-level labels indicating which points lie on contact fracture regions.
  - Contact Graph: A fragment-level adjacency matrix $A$ predicting which fragments are neighbors.
- Objective: The model is trained with a combined loss: $\mathcal{L} = \mathcal{L}_{rf}$ (flow matching) + $\lambda_F \mathcal{L}_F$ (fracture classification) + $\lambda_A \mathcal{L}_A$ (adjacency classification).
Pose Recovery:
- Once the assembled query points are generated, rigid transformations ( $T_i$ ) are recovered via post-processing rigid alignment between the input and output query points.

B. SARe-Refine: Inference-Time Correction

To address structural drift in large-scale assemblies, SARe introduces an optional refinement stage that uses the predicted structure to guide a second inference pass.

Geometric Verification:
- The initial assembly and predicted adjacency graph are analyzed.
- Candidate edges are verified using voxel-based geometric consistency checks (checking for interpenetration and fracture surface overlap).
- Unreliable edges are pruned, leaving a reliable subgraph (stable substructures).
RePaint-Style Resampling:
- The reliable substructures are "clamped" (fixed) to their first-pass positions.
- The uncertain regions are resampled in a second pass using a RePaint-style inpainting strategy. The model blends the stable region back into the trajectory at each timestep, effectively correcting errors in the uncertain parts while preserving verified correct parts.

3. Key Contributions

Explicit Contact Modeling: The paper identifies that explicit modeling of contact structures (adjacency graphs and fracture surfaces) is the key to scalability. It moves adjacency from an implicit signal to an actionable structural variable.
SARe-Gen Framework: A novel generative framework that jointly predicts poses and contact structures using query-point-based tokenization and Rectified Flow, avoiding the need for separate structural pretraining.
SARe-Refine Mechanism: A geometry-verified, inference-time correction module that anchors reliable substructures and resamples uncertain regions, significantly reducing error accumulation in large- $K$ scenarios.
Large-Scale Benchmark: The authors released a large-scale fracture reassembly benchmark derived from real-world scanned objects (OmniObject3D and Fantastic Breaks), providing ~55K training samples with structural supervision signals.

4. Experimental Results

The method was evaluated on three datasets: Breaking Bad (synthetic everyday/artifact objects), OmniObject3D (scanned real objects), and Fantastic Breaks (manually scanned complex objects).

State-of-the-Art Performance: SARe-Gen outperforms existing methods (including GARF, RPF, DiffAssemble, and Jigsaw) across all metrics (Part Accuracy, Chamfer Distance, Rotation/Translation Error).
- On Breaking Bad (Everyday), SARe achieved 94.98% Part Accuracy vs. 83.07% for the next best (RPF).
Scalability: As the fragment count ( $K$ ) increases, existing methods suffer from sharp drops in performance due to contact errors. SARe demonstrates graceful degradation, maintaining high accuracy even at $K=50$ .
Refinement Impact: SARe-Refine provides significant gains on "hard" cases (initial PA < 95%), improving Part Accuracy by 2.5% to 3.0% on average for difficult subsets.
Generalization: The model trained on joint data transfers robustly to out-of-domain datasets (OmniObject3D, Fantastic Breaks), indicating the learned structural cues are not dataset-specific.

5. Significance

Solving the Scalability Bottleneck: The paper addresses a critical limitation in current 3D assembly research: the inability of end-to-end models to scale to high fragment counts due to structural uncertainty.
Actionable Structure: By treating the contact graph as a predicted output rather than a byproduct, the method enables closed-loop correction, allowing the system to self-correct structural errors during inference.
Practical Application: The approach is highly relevant for digital heritage restoration (reconstructing shattered artifacts), robotic assembly, and accident scene reconstruction, where fragments are numerous, irregular, and lack semantic labels.

In summary, SARe represents a paradigm shift from purely pose-regression approaches to structure-aware generative modeling, proving that explicitly reasoning about how fragments connect is essential for solving large-scale 3D reassembly problems.