BrepCoder: A Unified Multimodal Large Language Model for Multi-task B-rep Reasoning

BrepCoder is a unified Multimodal Large Language Model that interprets industry-standard Boundary Representation (B-rep) data as structural code to enable a single model to perform diverse CAD tasks, such as completion, error correction, and question answering, through a two-stage training strategy focused on geometric feature learning and task generalization.

The Gist

Imagine you are a master architect who has spent a lifetime building incredible 3D structures. You have a library of blueprints, but they are written in a secret, complex code that only a few people understand.

For a long time, computers trying to learn from these blueprints had to be trained as specialists. If you wanted a computer to fix a broken wall, you built a "Wall-Fixer Robot." If you wanted to finish a half-built house, you built a "House-Completer Robot." These robots were great at their one job, but they couldn't talk to each other, and if you gave them a new task, they were useless.

BrepCoder is like a Universal Genius Apprentice that changes the game. Instead of being a specialist, it's a single, smart model that can do everything: fix errors, finish designs, answer questions, and even reverse-engineer a finished building back into its original blueprint.

Here is how it works, broken down into simple concepts:

1. The Language Problem: Speaking "Code" instead of "Pictures"

Most AI models look at 3D objects as clouds of dots (like a digital spray-paint job) or flat pictures. But in the real world of engineering, buildings are stored as B-reps (Boundary Representations). Think of a B-rep not as a picture, but as a mathematical recipe that defines exactly how every curve, edge, and surface connects.

The problem is that standard AI doesn't speak "Mathematical Recipe."

• The Old Way: Trying to teach the AI to recognize a dot-cloud. It's like trying to teach a chef to cook by showing them a blurry photo of the meal.
• The BrepCoder Way: The authors realized that CAD (Computer-Aided Design) is actually just code. They translated these complex 3D recipes into Python-like code (a language computers already know how to read).
  • Analogy: Instead of showing the AI a picture of a chair, they gave it the instruction manual: "Take a square, cut a hole here, and glue a leg there."

2. The Two-Stage Training: "Apprentice" then "Master"

The model learns in two distinct phases, like a student going to school.

Stage 1: The Reverse Engineering School (Learning the Logic)
Imagine giving the apprentice a finished, beautiful sculpture and asking them to write down the exact steps to build it from scratch.

• The model looks at the 3D shape (the B-rep) and tries to write the code (the Python instructions) that created it.
• By doing this thousands of times, the model doesn't just memorize shapes; it learns the logic of design. It understands why a curve exists and how a hole was cut. It internalizes the "grammar" of engineering.

Stage 2: The Multi-Task Intern (Applying the Logic)
Now that the model understands the logic, it can do anything:

• Completion: You give it the first half of a code (a half-built house), and it finishes the rest.
• Error Correction: You give it a code with a mistake (like a wall built in the wrong place), and it fixes the code to match the intended shape.
• CAD-QA: You ask, "How many windows are in this room?" and it counts them by "reading" the code, not just guessing from a picture.

3. Why is this a Big Deal?

• One Brain, Many Jobs: Before this, you needed a different AI for every task. Now, you have one "Universal Agent" that can switch hats instantly.
• It Speaks the Industry's Language: Because it uses the actual industry-standard format (B-rep) and translates it to code, it doesn't lose information. It's like speaking to a human in their native tongue rather than a broken translation.
• It's Smarter than "Guessing": Other models look at a 2D image and guess the 3D shape (which often leads to errors). BrepCoder looks at the structural "code" of the object, so it knows the exact dimensions and connections.

The Bottom Line

BrepCoder is a bridge between the messy, complex world of 3D engineering and the logical, powerful world of Large Language Models. By teaching the AI to see 3D designs as code, it has created a "General Purpose CAD Agent" that can think, reason, and build just like a human engineer, but much faster and without getting tired.

It's the difference between giving a robot a photo of a cake and asking it to bake one, versus giving it the recipe and letting it bake the perfect cake every time.

Technical Summary

1. Problem Statement

Current deep learning approaches in Computer-Aided Design (CAD) face three primary limitations:

• Task-Specific Fragmentation: Most existing models are designed for single tasks (e.g., reverse engineering, completion, or error correction) and require distinct architectural modifications for new tasks, hindering scalability and knowledge sharing.
• Suboptimal Input Modalities: While many models utilize point clouds or images, the industry standard for CAD data is Boundary Representation (B-rep). Existing methods often ignore B-rep or treat it merely as a static geometric shape rather than a procedural history.
• Lack of Design Logic Understanding: Current B-rep based models (e.g., BrepLLM) often align B-reps with natural language captions. This captures external geometric features but fails to understand the procedural design logic (the sequence of operations) required to construct the model, limiting their ability to perform complex engineering tasks like code generation or error correction.

2. Methodology: BrepCoder Framework

BrepCoder is a unified Multimodal Large Language Model (MLLM) that treats B-rep data as structural code. It employs a two-stage training strategy to align geometric features with Python-like CAD code.

A. Data Representation: Python-like CAD Code

Instead of using discrete integer tokens (as in DeepCAD), BrepCoder converts CAD modeling sequences into Python-like code (based on CAD-Llama). This representation includes:

• Objectification: 2D sketches are declared as unique variables.
• Method Invocation: Geometric commands (Line, Arc, Circle) are executed as method calls on sketch objects.
• Explicit Reference: 3D operations (e.g., Extrude) explicitly reference the sketch variables.
  This format leverages the pre-trained programming reasoning capabilities of Large Language Models (LLMs).

B. Architecture

• B-rep Encoder: Uses UV-Net to encode B-rep data. It converts the B-rep into a face-adjacency graph where nodes represent surfaces. Geometric information is projected onto 2D/1D grids and processed via a Graph Neural Network (GNN) to generate local and global feature embeddings.
• Multimodal Alignment (Phase 1): The framework adopts the CoCa (Contrastive Captioner) architecture.
  • It uses a cascaded attentional pooling mechanism to extract two types of features from the B-rep:
    1. Generation Features ($z_{gen}$): Fine-grained features used for code generation (captioning loss).
    2. Global Features ($z_{con}$): Summarized features used for global alignment (contrastive loss).
  • Loss Functions: The model is optimized using a weighted sum of Contrastive Loss (aligning B-rep and code globally) and Captioning Loss (generating code tokens from B-rep features).
• Multimodal Integration (Phase 2): The B-rep encoder is frozen. A Projector (MLP with GELU activation) maps the B-rep features into the LLM's embedding space. These are concatenated with text prompts and fed into the LLM (Qwen 2.5-1.5B).

C. Two-Stage Training Strategy

1. Stage 1: Reverse Engineering Pre-training:
   • Goal: Internalize the correlation between geometric features and design logic.
   • Task: The model takes a B-rep input and generates the corresponding Python-like CAD Code.
   • Outcome: The model learns the procedural steps required to construct a shape, effectively "learning to code" from geometry.
2. Stage 2: Multi-task Adaptation:
   • Goal: Apply learned logic to diverse downstream tasks.
   • Tasks:
     • Completion: Predicting the remainder of a CAD sequence given a partial code and B-rep.
     • Error Correction: Identifying and fixing logical contradictions or parameter errors in a CAD code based on the B-rep geometry.
     • CAD-QA: Answering questions about the design intent or properties of a model.

3. Key Contributions

1. Unified Framework: Proposes BrepCoder, the first unified MLLM that uses B-rep as the core input modality for diverse CAD tasks, eliminating the need for task-specific architectures.
2. Code-Based Alignment: Introduces a method to align B-rep data with Python-like CAD Code, enabling the LLM to utilize its robust programming reasoning for geometric interpretation.
3. Two-Stage Training: Demonstrates that pre-training on reverse engineering (learning design logic) significantly boosts performance on downstream tasks compared to direct fine-tuning.
4. Superior Generalization: Achieves state-of-the-art performance across reverse engineering, completion, error correction, and CAD-QA, proving the viability of B-rep as a modality for general-purpose CAD agents.

4. Experimental Results

The model was evaluated on the DeepCAD dataset (170K sequences) and the SGP-Bench for QA.

• Reverse Engineering (Stage 1):

  • Achieved a Chamfer Distance (CD) of 0.464 (scaled $10^{-3}$), a ~52% improvement over the previous state-of-the-art (CADCL at 0.972).
  • Achieved 82.01% parameter accuracy ($ACC_{param}$), indicating high precision in numerical values.
  • Maintained a very low Invalid Ratio (IR) of 0.86%, demonstrating high structural validity.
  • Note: While command accuracy ($ACC_{cmd}$) was slightly lower than some baselines (89.34% vs 90.31%), qualitative analysis showed this was due to modeling ambiguity (different valid command sequences producing identical shapes), not errors.

• Downstream Tasks (Stage 2):

  • Completion: Outperformed CAD-Llama with 92.69% command accuracy and 87.94% parameter accuracy. The B-rep modality provided richer topological information than text descriptions.
  • Error Correction: Significantly outperformed commercial MLLMs (GPT-4, Gemini) and ReCAD. BrepCoder achieved 99.08% command accuracy and 0.335 CD, whereas image-based models struggled with precise spatial coordinates and occlusion.
  • CAD-QA: Achieved 79% accuracy on SGP-Bench, comparable to much larger point-cloud-based models (PointLLM at 81%) despite using a smaller 1.5B parameter model.

• Ablation Studies:

  • Training the encoder from scratch (rather than using pre-trained segmentation weights) yielded better results.
  • The CoCa framework (captioning + contrastive) outperformed CLIP-only approaches.
  • Freezing the B-rep encoder during Stage 2 was crucial to preserve topological information.
  • Stage 1 pre-training was essential; removing it ("No Stage 1") caused performance drops across all downstream tasks.

5. Significance

BrepCoder represents a paradigm shift in CAD automation. By treating B-rep as a structural code and leveraging the reasoning capabilities of LLMs, it moves beyond simple shape recognition to procedural understanding. This unified approach allows a single model to handle the entire design lifecycle—from recovering design history to correcting errors and answering design questions—making it a strong candidate for a general-purpose CAD agent. The work bridges the gap between industrial standard data formats (B-rep) and modern generative AI, paving the way for more intelligent and automated design workflows.