ReConText3D: Replay-based Continual Text-to-3D Generation

1. Problem Definition

The paper addresses the gap in Continual Learning (CL) within the domain of Text-to-3D Generation. While current text-to-3D models (based on diffusion or flow objectives) can generate high-quality 3D assets from text, they are typically trained on static, curated datasets and then frozen. In real-world applications, models need to incrementally learn new object categories (e.g., new product lines or styles) without forgetting previously learned concepts.

The authors identify that naive fine-tuning of existing text-to-3D models on new data leads to catastrophic forgetting, where the model's ability to generate previously learned 3D assets degrades significantly. This is particularly challenging because the conditioning pathway (text encoder) and the generative pathway (3D decoder) co-evolve during fine-tuning, causing the shared conditioning manifold to drift and degrading alignment for old classes.

2. Methodology: ReConText3D

The authors propose ReConText3D, the first framework designed specifically for continual text-to-3D generation. It is a model-agnostic approach, meaning it can be integrated with any text-conditioned 3D generative backbone (e.g., diffusion-based or flow-based) without altering the underlying architecture.

The core mechanism is a Semantic Replay Strategy that constructs a compact, diverse memory of base-class examples to "rehearse" prior knowledge during the training of new classes. The process involves two main stages:

A. Replay Memory Construction

Instead of storing raw 3D meshes (which is memory-intensive), the method focuses on selecting representative text captions from the base dataset.

1. Count-Aware Budget Allocation: To handle the long-tailed distribution of real-world 3D datasets (where some classes have many samples and others few), the method allocates a replay quota for each class based on the square root of its sample count ($\sqrt{N}$). This prevents frequent classes from dominating the memory while ensuring underrepresented classes are still included.
2. Text-Embedding $k$-Center Selection: Within each class's allocated budget, the method selects specific captions using a $k$-center algorithm in the CLIP text-embedding space.
   • It uses the same CLIP encoder (ViT-L/14) employed by the generative models.
   • It greedily selects captions that maximize semantic diversity (coverage of the text-embedding manifold) within the class.
   • This ensures the replay memory exposes the model to varied textual conditions, preventing the model from overfitting to a narrow set of prompts during continual training.

B. Continual Training Process

During the novel-stage training:

• The model is trained on the new dataset ($D_{novel}$) mixed with the selected replay memory ($M_{replay}$).
• The replayed base prompts are used to anchor the text-to-shape mapping, mitigating the drift of the shared conditioning space.
• The underlying generative objective and architecture remain unchanged; only the training data composition is modified.

3. Benchmark: Toys4K-CL

To systematically evaluate this new setting, the authors introduce Toys4K-CL, a novel benchmark derived from the Toys4K subset of the TRELLIS-500K dataset.

• Structure: It consists of 90 well-populated classes split into Base (45 classes) and Novel (45 classes) sets.
• Design Goals:
  • Balanced & Diverse: Ensures comparable numbers of classes and assets across stages.
  • Adversarial Splits: Closely related categories (e.g., dog, cat, fox) are distributed across the Base and Novel sets to maximize interference and stress-test the model's ability to distinguish concepts.
  • Long-Tailed: Mimics real-world data distributions with class imbalance.

4. Key Contributions

1. First Framework: Establishes the first continual learning framework for text-to-3D generation.
2. Forgetting Analysis: Empirically demonstrates that state-of-the-art text-to-3D models suffer severe catastrophic forgetting under class-incremental training.
3. Novel Replay Method: Proposes a hybrid replay strategy combining count-aware budget allocation and text-embedding $k$-center selection to build compact, semantically diverse memories.
4. Benchmark: Introduces Toys4K-CL, the first benchmark for continual text-to-3D generation with balanced and adversarial splits.
5. Model Agnosticism: Validates the approach across different generative backbones (Flow-based and Diffusion-based).

5. Experimental Results

The authors evaluated ReConText3D on two representative backbones: TRELLIS-XL (Flow-based) and Shap-E (Diffusion-based).

• Quantitative Performance:
  • Forgetting Reduction: ReConText3D significantly reduces forgetting compared to naive fine-tuning. On TRELLIS-XL, it reduced CLIP forgetting by ~77% and FD (Fréchet Distance) forgetting by ~50%. On Shap-E, it reduced CLIP forgetting by ~56%.
  • Overall Quality: The method not only preserves base-class performance but often improves overall generation quality for both old and new classes, suggesting a "semantic rehearsal" effect that stabilizes training dynamics.
  • Comparison: It outperforms baselines like L2-SP (regularization) and standard fine-tuning, approaching the performance of "Joint Training" (training on all data at once, which is the theoretical upper bound).
• Qualitative Results:
  • Visual comparisons show that naive fine-tuning often confuses base classes with novel ones (e.g., generating a lizard instead of a dragon) or degrades geometry.
  • ReConText3D successfully retains semantic understanding and structural details for old classes while synthesizing plausible novel objects.
• Ablation Studies:
  • Memory Size: Performance improves steadily with larger replay buffers, but a compact memory (20% of the novel set size) is sufficient for stable generation.
  • Selection Strategy: The proposed $k$-center selection significantly outperforms random replay, highlighting the importance of semantic diversity in the replay set.

6. Significance

This work opens a new direction for incremental 3D generative modeling. By solving the catastrophic forgetting problem in text-to-3D, ReConText3D enables practical applications where 3D asset libraries must evolve over time (e.g., in game development, AR/VR, and robotics) without requiring retraining from scratch. The introduction of the Toys4K-CL benchmark provides a standardized platform for future research in continual 3D generation.