Technical Summary: Arbor: Explicit Geometric Conditioning for Controllable 3D Asset Generation

Problem Statement

Current text and image-conditioned 3D generation methods produce convincing assets but lack direct control over the specific spatial regions an object should occupy, avoid, or contact. In professional workflows (e.g., game production), artists often know spatial requirements—such as a chair fitting a specific seating envelope, a prop leaving clearance for motion, or a part exposing a contact surface—before generation begins. However, prompts and image views are poor carriers for these constraints, leading to a "slot machine" workflow where artists iterate through many prompts and perform manual cleanup. Existing 3D control methods typically focus on editing existing assets, completing partial geometry, or steering inference at sampling time, but they do not offer a native interface for specifying dense, typed local geometric requirements (hull, avoidance, touch) during the generation of a full object from a text prompt.

Methodology

Arbor introduces a trainable attachment for text-conditioned latent 3D generation that converts explicit constraint meshes into a native control signal. The system operates on a frozen text-conditioned backbone (specifically the TRELLIS family, using TRELLIS 1 for generation and TRELLIS 2 for encoding) and injects geometric constraints without modifying the backbone's weights.

1. Constraint Representation

Arbor defines three types of spatial constraints, specified by the user as 3D meshes:

• Hull Regions: Volumes where generated geometry must exist (e.g., a seat).
• Avoidance Regions: Volumes that must remain empty (e.g., clearance space).
• Touch Regions: Surfaces the object must contact without extending significantly past them (e.g., legs meeting the ground).

2. Encoding Pipeline

To integrate these dense meshes into the compressed latent space of the generator:

• Encoder Repurposing: Arbor utilizes the frozen encoder stack of TRELLIS 2 (T2). While T2 is designed for image-conditioned generation, its encoder is repurposed to process constraint meshes.
• Signal Injection: The constraint meshes are converted into aligned shape and material fields on a voxel grid. Crucially, the auxiliary attribute channels (originally for material parameters) are repurposed to encode binary flags for the constraint types (hull, avoid, touch) alongside surface normals.
• Token Generation: The frozen T2 encoders map these fields into compact, sparse 32-dimensional tokens. These tokens are projected to match the width of the T1 generator and augmented with learned 3D positional embeddings.

3. Geometry Router and Adapter

A central challenge is injecting dense constraint tokens into the sparse latent structure of the generator efficiently. Arbor solves this via a Geometry Router and a Residual Adapter:

• Routing: The generator's latent query lattice (16³ grid) is partitioned into groups. For each group, the router selects the most relevant local constraint tokens based on spatial proximity (Top-K selection using anchor points) and appends a set of learned global summary tokens. This ensures each latent region receives only the constraint evidence relevant to its spatial location.
• Adapter Injection: A lightweight residual branch is inserted into each transformer block of the frozen generator (after text cross-attention but before the Feed-Forward Network). The adapter performs cross-attention between the text-conditioned hidden states and the routed geometry context, injecting the constraint signal as a residual update.
• Training: Only the adapter, router, projection layers, and summary modules are trained. The backbone (T1) and encoders (T2) remain frozen. The model is optimized using the standard flow-matching objective of the backbone, with no explicit constraint compliance losses used in the primary training setup.

Key Contributions

1. Explicit Geometric Control Interface: A unified system for specifying typed regions (hull, touch, avoidance) directly as 3D meshes, allowing artists to co-author the generation process.
2. Encoded Geometry as Condition: Demonstration that frozen geometric encoders can be repurposed to transform constraint meshes and typed signals into compact latent tokens that preserve local structure and serve as a model input.
3. Geometry Router and Adapter: A novel architecture that assigns local constraint evidence to specific latent regions and injects this evidence into a frozen 3D generator via a routed residual branch, enabling dense geometric control without compromising the generator's variability.

Evaluation and Results

The authors evaluated Arbor on the Toys4K dataset using both procedurally generated (automatic) and hand-authored (manual) constraint benchmarks.

• Constraint Adherence: Arbor significantly outperforms baselines (including the base TRELLIS model, sampling-time steering methods like Gradient and SpaceControl, and completion-based methods like Spice-E) in adhering to hull, avoidance, and touch constraints.
• User Preference: In a pairwise user study (27 participants, 404 trials), the Arbor variants were preferred over the base TRELLIS model and other baselines. The standard Arbor model won 45.9% of comparisons, rising to 59.2% when merging all Arbor variants.
• Quality and Variation: Unlike sampling-time steering methods which often degrade geometry quality or collapse shapes, Arbor maintains readable object structures and preserves the generative variation of the underlying prior.
• Metrics: Arbor achieved high scores on "Control Score" (a harmonic mean of constraint adherence, volume match, and visual similarity), demonstrating it can satisfy constraints without overfilling or losing the prompt's semantic intent.
• Ablations: Experiments confirmed that local routing is the primary mechanism for constraint adherence (retaining 84.3% of IoU without global summaries), though global summaries provide necessary object-level context. The signal stream (typed flags) was found to be critical; removing it caused a significant drop in performance.

Significance and Claims

The paper claims that Arbor provides a missing interface for 3D asset generation, allowing for explicit volumetric control without interfering with the generator's shape creation variability. By treating constraints as typed, local requirements rather than target evidence or global priors, Arbor enables artists to define specific spatial behaviors (e.g., "leave room for motion" or "expose a contact surface") directly.

The authors position Arbor as a step toward making 3D asset generation more reliable for production use, where animations and interactions require accurate interaction surfaces. They note that while the current system successfully guides the sparse structure stage, it does not yet control later refinement stages (surface detail and materials) or fully resolve semantic conflicts between constraints and prompts (e.g., ensuring a volume is functionally a "seat"). The work suggests future directions involving richer part labels and extending the routed attachment to later generation stages.