Engineering Regression Without Real-Data Training: Domain Adaptation for Tabular Foundation Models Using Multi-Dataset Embeddings

This paper introduces TREDBench and an embedding-guided synthetic data curation method to adapt the TabPFN 2.5 foundation model for engineering regression tasks, achieving superior predictive accuracy and data efficiency without requiring training on real-world engineering samples.

The Gist

The Big Problem: The "Data Starvation" of Engineers

Imagine you are an engineer trying to design a new, super-efficient car engine. To do this, you usually need to run thousands of computer simulations or build physical prototypes. But these are incredibly expensive and slow. You might only have data from 12 crash tests or 50 wind tunnel runs.

In the world of Artificial Intelligence (AI), this is a disaster. Modern AI models (like the ones that write poetry or recognize cats) are like giant, hungry elephants. They need to eat millions of data points to learn. If you feed an elephant a single crumb (your 12 crash tests), it won't learn anything.

Traditionally, engineers have had to build a tiny, custom AI model for every single problem. It's like hiring a different chef for every single meal you want to cook. It's slow, expensive, and inefficient.

The New Hope: The "Universal Chef" (Foundation Models)

Recently, scientists created "Foundation Models" (like TabPFN). Think of these as Universal Chefs. Instead of learning to cook one specific dish, they have been trained on millions of fake recipes generated by a computer. They have learned the general rules of cooking (how heat affects food, how spices mix, etc.).

The hope was: Can we take this Universal Chef, who has only cooked with fake ingredients, and have them cook a real engineering meal perfectly?

The Discovery: The "Fake vs. Real" Gap

The authors of this paper asked a crucial question: Do the fake ingredients the chef learned on actually taste like real engineering data?

To find out, they built a massive library called TREDBench. They gathered 83 different datasets:

• 35 Engineering Datasets: Real data about cars, bridges, and materials.
• 48 Non-Engineering Datasets: Real data about house prices, wine quality, and sports stats.
• Synthetic Data: The "fake" data the AI was originally trained on.

They used a special tool (a "dataset embedding") to visualize these datasets. Imagine a map where every dataset is a dot.

• The Result: The "Real Engineering" dots lived in their own neighborhood. The "Real Non-Engineering" dots lived in a different neighborhood. But the "Fake Synthetic" dots? They were mostly scattered in a chaotic wasteland, far away from the engineering neighborhood.

The Analogy: It's like training a chef on a diet of only plastic food. Even though the plastic food looks like a burger, it doesn't have the texture or taste of a real burger. When the chef tries to cook a real burger, they fail because the "plastic training" didn't prepare them for the reality of meat and buns.

The Solution: The "Taste-Test" Filter

The team realized they couldn't just feed the chef more plastic food. They needed to find the rare pieces of plastic food that actually tasted like a real burger.

They came up with a clever, two-step plan that did not require any real engineering data (which is the whole point, since real data is scarce):

1. Generate a Mountain of Fake Data: They created 10,000 new synthetic datasets.
2. The "Taste-Test" (Embedding): They used the AI's own internal "sense of smell" to look at these 10,000 datasets. They asked the AI: "Which of these fake datasets looks most like the engineering neighborhood on our map?"
3. The Selection: They picked the top 200 "fake" datasets that were the closest match to real engineering data.
4. The Fine-Tuning: They took the Universal Chef and gave them a short, intensive training course using only those 200 selected fake datasets.

The Magic: They never showed the chef a single real engineering data point during this training. They just taught the chef to recognize the style of engineering data using the best possible fake examples.

The Results: A Super-Chef is Born

When they tested this new, "fine-tuned" chef on real engineering problems:

• It was faster: It needed 1.75 times less data to get the same answer as the original model.
• It was more accurate: It beat the original model in 29 out of 35 engineering problems.
• It beat the competition: It even beat the current industry leader (AutoGluon) in 27 out of 35 problems.

Why This Matters

This paper proves that we don't need to wait for engineers to generate millions of dollars worth of real data to build great AI.

Instead, we can use principled curation. Think of it like a music producer. They don't need to record a million real bands to find a hit song. They can use a synthesizer to generate millions of sounds, use a smart filter to find the ones that sound like a "rock band," and train their AI on those.

The Takeaway:
By carefully selecting the right kind of "fake" data, we can turn generic AI models into specialized experts for engineering, solving the "data starvation" problem without needing to collect more real-world data. We are essentially teaching the AI to dream in the right language so it can speak it fluently when the time comes.

Technical Summary

1. Problem Statement

Predictive modeling in engineering is historically constrained by data scarcity. Engineering datasets (e.g., from simulations or physical experiments) are often small, expensive to generate, and siloed. This prevents the application of Foundation Models, which typically require massive, diverse datasets for pre-training.

While Tabular Foundation Models (specifically TabPFN) have shown promise by pre-training on millions of procedurally generated synthetic datasets, a critical gap exists:

• The Synthetic-Real Domain Gap: The statistical structure of procedurally generated data often differs significantly from real-world engineering data.
• Performance Variability: TabPFN performs well on some engineering tasks but poorly on others (e.g., the welded beam design problem), suggesting that standard synthetic generators do not fully capture the nuances of engineering domains.
• The Challenge: How can we adapt a foundation model trained only on synthetic data to perform optimally on real engineering tasks without using any real engineering data for fine-tuning (due to scarcity or privacy constraints)?

2. Methodology

The authors propose a framework called TREDBench and an Embedding-Guided Synthetic Data Curation pipeline to bridge the domain gap.

A. TREDBench: A New Benchmark

The authors curated TREDBench, a collection of 83 tabular regression datasets:

• 35 Engineering Datasets: Covering mechanical, material, chemical, and electrical engineering (e.g., vehicle crash tests, aerodynamic drag, heat exchangers).
• 48 Non-Engineering Datasets: Covering economics, real estate, and logistics.
• Standardization: All datasets were processed to have 1,024 samples (via random duplication for smaller sets) and standardized features/labels.

B. Dataset Embeddings & Domain Analysis

The authors utilized TabPFN 2.5 to create a "dataset-level embedding" space:

1. Embedding Construction: For each dataset, TabPFN 2.5 processes the data points during in-context inference. The mean of these point-level embeddings is calculated to create a single 192-dimensional vector representing the entire dataset's statistical structure.
2. Distinguishability Analysis: An XGBoost classifier was trained to distinguish between dataset types in this embedding space.
   • Engineering vs. Non-Engineering: Distinguishable with 65.8% accuracy.
   • Procedural Synthetic vs. Engineering: Highly distinguishable with 96.8% accuracy, confirming a significant domain gap.
   • Visual Insight: T-SNE visualization showed that while most synthetic data clusters away from engineering data, a small subset overlaps, suggesting some synthetic generators produce "engineering-like" distributions.

C. Embedding-Guided Synthetic Data Curation

To adapt the model without real data, the authors proposed a selection pipeline:

1. Generation: Generate 10,000 new procedural synthetic regression datasets.
2. Embedding: Compute the 192-dimensional embedding for each synthetic dataset using TabPFN 2.5.
3. Classification: Train a binary classifier (XGBoost) to distinguish between the 35 real engineering embeddings and a sample of synthetic embeddings.
4. Selection: Apply the classifier to the remaining 9,750 synthetic datasets. Select the top 200 datasets with the highest predicted probability of being "engineering-like."
5. Result: This sub-selection reduced the distinguishability between synthetic and engineering data from 96.8% to 88.9%, effectively narrowing the domain gap.

D. Synthetic-Only Adaptation

• Fine-Tuning: The selected 200 "engineering-like" synthetic datasets were used to perform continued pre-training (fine-tuning) of TabPFN 2.5 for 5 epochs.
• Constraint: No real engineering data was used during this fine-tuning process. The model weights were updated solely based on the curated synthetic distribution.

3. Key Contributions

1. TREDBench: The first curated benchmark specifically for tabular regression in engineering, featuring 83 datasets with expert-verified labels.
2. Quantification of the Domain Gap: Demonstrated via dataset embeddings that standard procedural generators are highly distinguishable from real engineering data, validating the need for adaptation.
3. Embedding-Guided Curation: A novel method to filter synthetic data based on its proximity to the target domain in a foundation model's latent space, effectively turning a generic generator into a domain-specific "data engine."
4. Synthetic-Only Adaptation: Proved that a foundation model can be significantly improved for a specific domain (engineering) without ever seeing a single real sample from that domain during the adaptation phase.

4. Experimental Results

The adapted model was evaluated on 35 engineering regression datasets against two baselines:

1. Original TabPFN 2.5 (pre-trained on generic synthetic data).
2. AutoGluon (state-of-the-art AutoML ensemble).

Performance Metrics:

• Accuracy Wins:
  • Outperformed original TabPFN 2.5 on 29/35 (82.9%) datasets.
  • Outperformed AutoGluon on 27/35 (77.1%) datasets.
• Data Efficiency:
  • vs. TabPFN 2.5: Achieved a mean multiplicative data-efficiency gain of 1.75x (the adapted model needed ~1.75x less data to reach the same performance).
  • vs. AutoGluon: Achieved a mean multiplicative data-efficiency gain of 4.44x.
• Consistency: While performance gains were widespread, they were not uniform; the model underperformed on 6 datasets compared to the base TabPFN and 8 compared to AutoGluon, indicating room for further refinement in the generation process.

5. Significance and Implications

• Solving Data Scarcity: This work provides a viable pathway for deploying foundation models in data-sparse scientific and industrial domains where collecting real data is the primary bottleneck.
• Paradigm Shift: It moves engineering predictive modeling away from bespoke, task-specific models toward reusable, generalizable systems that can adapt to new design problems with minimal data.
• Synthetic Data as a "Data Engine": The paper demonstrates that procedural generators are not just noise; with principled curation (guided by foundation model embeddings), they can be transformed into high-fidelity, domain-relevant training sources.
• Future Directions: The authors suggest that future work could refine the generation procedures themselves (not just selection) and expand these methods to other data modalities beyond tabular regression.

In summary, the paper establishes that domain adaptation via synthetic data curation is a powerful strategy for engineering AI, enabling foundation models to achieve state-of-the-art performance on real-world engineering problems without requiring real-world training data.