Technical Summary: MeshFIM

Problem Statement

Autoregressive (AR) models have demonstrated the ability to generate high-quality, artist-grade low-poly meshes from point clouds. However, existing AR generators operate in an "all-or-nothing" manner: to modify a single local region, the entire mesh must be regenerated from scratch. This approach is computationally wasteful and disrupts satisfactory topology in unaffected areas. Furthermore, classical local editing techniques (e.g., hole filling, subdivision) often fail to preserve topological coherence or design intent in low-poly settings, where mesh structure encodes specific semantic information rather than just surface discretization.

The core challenge in adapting AR models for local editing lies in the unique geometric and topological constraints of 3D meshes:

1. Serialization Ordering: Meshes are 2D complexes embedded in 3D space. When serialized into a flat token sequence for AR generation, spatially contiguous neighborhoods lose their local ordering, making it difficult for models to preserve edge flow and density transitions.
2. Exact Boundary Attachment: Generated patches must reuse exposed boundary vertices exactly. Even sub-quantization coordinate mismatches result in visible cracks or T-junctions.
3. Generation Overflow: Without strict constraints, generated faces may spill beyond the intended target region.
4. Train-Inference Gap: Training contexts are extracted from clean ground-truth meshes, whereas inference contexts may contain artifacts from prior steps or user selections, leading to robustness issues.

Methodology: MeshFIM

The authors introduce MeshFIM, a Fill-in-the-Middle (FIM) framework adapted from code completion models to autoregressive mesh generation. MeshFIM treats local mesh editing as a conditional infilling task, where a target region $B$ is regenerated based on a surrounding context $A\setminus B$ and a reference geometry $P$.

The framework employs five complementary design choices to address the specific challenges of mesh editing:

1. Boundary Vertex Markers

To enforce exact attachment, the model explicitly marks context tokens corresponding to vertices on the exposed boundary ($V_\partial$). These embeddings are modified by adding a learned marker vector ($m_{bdry}$). This provides an explicit cue to the decoder regarding which context vertices must be reused as stitching anchors, ensuring seamless connectivity.

2. Expanded Context Width

To prevent generation overflow (faces spilling beyond the target region), the context window is expanded. Instead of a minimal neighborhood, the model utilizes a wider context (e.g., three breadth-first rings). This exposes more surrounding geometry, providing sufficient evidence for the model to infer local curvature and edge flow, thereby keeping the generation localized.

3. Context Positional Embeddings

Since serialization flattens the mesh, the model risks treating the context as an unordered bag of vertices. To restore local topological structure, learned positional embeddings are added to the serialized context sequence. This helps the model preserve surrounding patterns such as edge flow and face density.

4. Low-Poly Geometry Encoder with Gated Subtraction

Standard point-cloud conditioners encode the reference surface but lack knowledge of what the existing low-poly mesh already covers. MeshFIM introduces a mechanism to bridge this:

• A shared Perceiver-IO encoder encodes both the reference point cloud ($P_{gt}$) and the existing low-poly mesh ($P_{lp}$, with the target region removed) using the same set of query positions.
• A lightweight gate network produces a vector $g$ based on the difference between the aligned latents ($z_{gt}$ and $z_{lp}$).
• The final conditioning signal is an attenuation of the reference latent: $\hat{z} = (1 - g) \cdot z_{gt}$.
• Mechanism: Where the existing mesh already covers the reference surface well, the gate suppresses the reference signal. Where the mesh is missing (the hole), the gate allows the full reference latent to pass, directing generation specifically toward the missing region.

5. Context Augmentation

To close the train-inference gap, the training process includes context augmentation. Context vertex coordinates are perturbed with random offsets, and boundary vertices are similarly perturbed. This exposes the model to noisy and incomplete contexts during training, improving robustness during inference.

Key Contributions

• First Local Low-Poly Mesh Editor: MeshFIM is the first framework to enable accurate, local editing of low-poly meshes using autoregressive generation, avoiding the need for full regeneration.
• FIM Adaptation for 3D: It successfully adapts the Fill-in-the-Middle paradigm to 3D meshes, addressing non-trivial geometric and topological constraints unique to 3D data.
• Novel Architectural Components: The paper introduces specific mechanisms (boundary markers, gated subtraction, context augmentation) that collectively solve the problems of boundary matching, overflow, and topological coherence.
• Dual Applications: The framework supports both interactive, brush-based editing and an automatic defect repair pipeline.

Experimental Results

The authors evaluated MeshFIM against a range of baselines, including classical hole filling, neural subdivision, and whole-mesh generation with stitch-back schemes.

• Quantitative Performance: MeshFIM significantly outperforms baselines in boundary correctness (Average Vertex Matching Ratio of 99.75% vs. <34% for others) and geometric quality (Chamfer Distance Positive Rate of 95.54%). It also demonstrates superior locality, with minimal impact on surrounding geometry.
• Ablation Studies: Detailed ablations confirm the necessity of each component. Boundary vertex markers provided the largest single gain in perfect matching ratio. Expanding the context width sharply reduced overflow rates. The low-poly geometry encoder further suppressed overflow and improved generation quality.
• Applications:
  • Interactive Editing: A web-based interface allows users to paint target regions, which are then regenerated seamlessly.
  • Automatic Repair: A pipeline using graphics heuristics detects defects (e.g., missing faces, flipped normals) and iteratively repairs them using MeshFIM, achieving an 88.9% repair rate on damaged meshes after four rounds.

Significance and Claims

The paper claims that MeshFIM represents a significant step forward in practical 3D modeling workflows. By enabling local edits without destroying existing topology, it aligns with the iterative nature of professional modeling. The authors emphasize that their method is the first to combine the high-quality topology generation of AR models with the precision required for local editing, effectively bridging the gap between generative AI and practical mesh manipulation. The work suggests that future 3D asset creation can move away from "regenerate everything" approaches toward targeted, context-aware refinement.